Use RESTful API

Kylin 4.0 支持的 REST API 及其用法在 Apache Kylin Wiki 页面列出：

Build Cube with API

1. Authentication

 Currently, Kylin uses basic authentication.

 Add Authorization header to first request for authentication
 Or you can do a specific request by POST
[Link]
 Once authenticated, client can go subsequent requests with cookies.

POST [Link]

Authorization:Basic xxxxJD124xxxGFxxxSDF
Content-Type: application/json;charset=UTF-8
2. Get details of cube.

 GET [Link]
cubeName={cube_name}&limit=15&offset=0
 Client can find cube segment date ranges in returned cube detail.

GET [Link]
cubeName=test_kylin_cube_with_slr&limit=15&offset=0

Authorization:Basic xxxxJD124xxxGFxxxSDF
Content-Type: application/json;charset=UTF-8
3. Then submit a build job of the cube.

 PUT [Link]
 For put request body detail please refer to Build Cube API.
o startTime and endTime should be utc timestamp.
o buildType can be BUILD ,MERGE or REFRESH. BUILD is for building a new
segment, REFRESH for refreshing an existing segment. MERGE is for merging
multiple existing segments into one bigger segment.
 This method will return a new created job instance, whose uuid is the unique id of job to
track job status.

PUT
[Link]

Authorization:Basic xxxxJD124xxxGFxxxSDF
Content-Type: application/json;charset=UTF-8

{
"startTime": 0,
 "endTime": 1388563200000,
"buildType": "BUILD"
}
4. Track job status.

 GET [Link]
 Returned job_status represents current status of job.

5. If the job got errors, you can resume it.

 PUT [Link]

6. Adjust the cuboid list of a cube and trigger optimize segment job

 PUT [Link]

Backup Metadata
Kylin organizes all of its metadata (including cube descriptions and instances, projects,
inverted index description and instances, jobs, tables and dictionaries) as a hierarchy file
system. However, Kylin uses mysql to store it, rather than normal file system. If you check
your kylin configuration file([Link]) you will find such a line:

## The metadata store in mysql

[Link]=kylin_metadata@jdbc,driverClassName=[Link]
ver,url=jdbc:mysql://localhost:3306/kylin_database,username=,password=

This indicates that the metadata will be saved as a table called kylin_metadata in mysql
database kylin_database.

Metadata directory

Kylin metastore use resource root path + resource name + resource suffix as
key to store metadata. You can refer to the following table to use ./bin/[Link].

Resource root path resource name resource suffix

/cube /cube name .json

/cube_desc /cube name .json

/cube_statistics /cube name/uuid .seq

/model_desc /model name .json

/project /project name .json

/table /[Link]–project name .json

Resource root path resource name resource suffix

/table_exd /[Link]–project name .json

/execute /job id  

/execute_output /job id-step index  

/user /user name  

View metadata

If you want to view some metadata, you can run:

./bin/[Link] list /path/to/store/metadata

to list the entity stored in specified directory, and then run:

./bin/[Link] cat /path/to/store/entity/metadata.

to view one entity metadata.

Backup metadata with binary package

Sometimes you need to backup the Kylin’s metadata store from mysql to your disk file
system.
In such cases, assuming you’re on the hadoop CLI(or sandbox) where you deployed Kylin,
you can go to KYLIN_HOME and run :

./bin/[Link] backup

to dump your metadata to your local folder a folder under

KYLIN_HOME/metadata_backps, the folder is named after current time with the syntax:
KYLIN_HOME/meta_backups/meta_year_month_day_hour_minute_second

In addition, you can run:

./bin/[Link] fetch /path/to/store/metadata

to dump metadata selectively. For example, run ./bin/[Link] fetch

/cube_desc/ to get all cube desc metadata, or run ./bin/[Link] fetch
/cube_desc/kylin_sales_cube.json to get single cube desc metadata.
Restore metadata with binary package

In case you find your metadata store messed up, and you want to restore to a previous
backup:

Firstly, reset the metadata store (this will clean everything of the Kylin metadata store in
mysql, make sure to backup):

./bin/[Link] reset

Then upload the backup metadata to Kylin’s metadata store:

./bin/[Link] restore
$KYLIN_HOME/meta_backups/meta_xxxx_xx_xx_xx_xx_xx
Restore metadata selectively (Recommended)

If only changes a couple of metadata files, the administrator can just pick these files to
restore, without having to cover all the metadata. Compared to the full recovery, this
approach is more efficient, safer, so it is recommended.

Create a new empty directory, and then create subdirectories in it according to the location
of the metadata files to restore; for example, to restore a Cube instance, you should create a
“cube” subdirectory：

mkdir /path/to/restore_new
mkdir /path/to/restore_new/cube

Copy the metadata file to be restored to this new directory:

cp meta_backups/meta_2016_06_10_20_24_50/cube/kylin_sales_cube.json
/path/to/restore_new/cube/

At this point, you can modify/fix the metadata manually.

Restore from this directory:

cd $KYLIN_HOME
./bin/[Link] restore /path/to/restore_new

Only the files in the folder will be uploaded to Kylin metastore. Similarly, after the
recovery is finished, click Reload Metadata button on the Web UI to flush cache.

Backup/restore metadata in development env

When developing/debugging Kylin, typically you have a dev machine with an IDE, and a
backend sandbox. Usually you’ll write code and run test cases at dev machine. It would be
troublesome if you always have to put a binary package in the sandbox to check the
metadata. There is a helper class called SandboxMetastoreCLI to help you
download/upload metadata locally at your dev machine. Follow the Usage information and
run it in your IDE.

Cleanup Storage
Wiki :
[Link]
+4

Optimize Build and Query

Kylin 4 is a major architecture upgrade version, both cube building engine and query
engine use spark as calculation engine, and cube data is stored in parquet files instead of
[Link] the build/query performance tuning is very different from Kylin 3 tuning.

About the build/query performance tuning of Apache Kylin4.0, Please refer to:
How to improve cube building and query performance of Apache Kylin4.0.

At the same time, you can refer to kylin4.0 user’s optimization practice blog:
why did Youzan choose Kylin4

Config different spark Pool for different

types of SQL
Please check document: Use different spark pool for different query

Upgrade From Old Versions

Compared with Kylin 3.x and previous versions, Kylin 4.0’s storage engine has changed
from HBase to Parquet. Therefore, if you need to upgrade from Kylin 3.x and previous
versions to kylin4.0, the built cuboid data can’t be upgraded, you can only upgrade
metadata.

Please refer to : How to migrate metadata to Kylin 4

Use Utility CLIs

Kylin has some client utility tools. This document will introduce the following class:
[Link], [Link], [Link],
[Link] and [Link]. Before using these tools, you
have to switch to the KYLIN_HOME directory.

[Link]
Function

[Link] outputs the value of Kylin properties.

How to use

After the class name, you can only write one parameter, conf_name which is the parameter
name that you want to know its value.

./bin/[Link] [Link] <conf_name>

For example:

./bin/[Link] [Link] [Link]

Result:

all

If you do not know the full parameter name, you can use the following command, then all
parameters prefixed by this prefix will be listed:

./bin/[Link] [Link] <prefix>.

For example:

./bin/[Link] [Link] [Link].

Result:

max-concurrent-jobs=10
retry=3
sampling-percentage=100
[Link]
Function

[Link] is to extract Cube related info for debugging / distributing

purpose.

How to use

At least two parameters should be followed.

./bin/[Link] [Link] -<conf_name>
<conf_value> -destDir <your_dest_dir>

For example:

./bin/[Link] [Link] -cube

kylin_sales_cube -destDir /tmp/kylin_sales_cube

Result:
After the command is executed, the cube, project or hybrid you want to extract will be
dumped in the specified path.

All supported parameters are listed below:

Parameter Description

allProjects Specify realizations in all projects to extract

compress Specify whether to compress the output with zip. Default true.

cube Specify which Cube to extract

destDir (Required) Specify the dest dir to save the related information

hybrid Specify which hybrid to extract

includeJobs Set this to true if want to extract job info/outputs too. Default false

Set this to true if want to extract segment details too, such as dict,
includeSegmentDetails
tablesnapshot. Default false

includeSegments Set this to true if want extract the segments info. Default true

onlyOutput When include jobs, only extract output of job. Default true

packagetype Specify the package type

project Which project to extract

   

[Link]
Function

[Link] is to ingest the extracted cube meta data into another metadata
store. It only supports ingest cube now.

How to use

At least two parameters should be specified. Please make sure the cube you want to ingest
does not exist in the target project.
Note: The zip file must contain only one directory after it has been decompressed.

./bin/[Link] [Link] -project

<target_project> -srcPath <your_src_dir>

For example:

./bin/[Link] [Link] -project querytest

-srcPath /tmp/newconfigdir1/[Link]

Result:
After the command is successfully executed, the cube you want to ingest will exist in the
srcPath.

All supported parameters are listed below:

Parameter Description

Skip the target Cube, model and table check and ingest by force. Use in caution
forceIngest because it might break existing cubes! Suggest to backup metadata store first.
Default false.

If table meta conflicts, overwrite the one in metadata store with the one in
overwriteTables srcPath. Use in caution because it might break existing cubes! Suggest to backup
metadata store first. Default false.

project (Required) Specify the target project for the new cubes.

srcPath (Required) Specify the path to the extracted Cube metadata zip file.

[Link]
Function

Apache Kylin have provided migration tool to support migrating metadata across different
clusters since version 2.0. Recently, we have refined and added new ability to
CubeMigration tool, The list of enhanced functions is showed as below:
- Support migrating all cubes in source cluster
- Support migrating a whole project in source cluster
- Support migrating and upgrading metadata from older version to Kylin 4

How to use

Please check: How to migrate metadata to Kylin4

Secure with LDAP and SSO

Enable LDAP authentication

Kylin supports LDAP authentication for enterprise or production deployment; This is

implemented with Spring Security framework; Before enable LDAP, please contact your
LDAP administrator to get necessary information, like LDAP server URL,
username/password, search patterns;

Configure LDAP server info

Firstly, provide LDAP URL, and username/password if the LDAP server is secured; The
password in [Link] need be encrypted; You can run the following command to get
the encrypted value:

cd $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/lib
java -classpath kylin-server-base-\<versioin\>.jar:kylin-core-
common-\<versioin\>.jar:[Link]:spring-core-
[Link]:[Link]
[Link] AES
<your_password>

Config them in the conf/[Link]. When you use the customized CA certificate
library for user authentication based on LDAPs, you need to configure
‘[Link]-truststore’, the value of this configuration will be added to
the JVM parameter [Link]:

[Link]-server=ldap://<your_ldap_host>:<port>
[Link]-username=<your_user_name>
[Link]-password=<your_password_encrypted>
[Link]-
truststore=<your_customized_CA_certificate_library>

Secondly, provide the user search patterns, this is by LDAP design, here is just a sample:

[Link]-search-base=OU=UserAccounts,DC=mycompany,DC=com
[Link]-search-pattern=(&(cn={0})(memberOf=CN=MYCOMPANY-
USERS,DC=mycompany,DC=com))
[Link]-group-search-base=OU=Group,DC=mycompany,DC=com

If you have service accounts (e.g, for system integration) which also need be authenticated,
configure them in [Link]-.*; Otherwise, leave them be empty;

Configure the administrator group

To map an LDAP group to the admin group in Kylin, need set the
“[Link]-role” to the LDAP group name (shall keep the original case), and
the users in this group will be global admin in Kylin.

For example, in LDAP the group “KYLIN-ADMIN-GROUP” is the list of administrators,

here need set it as:
[Link]-role=KYLIN-ADMIN-GROUP

Attention: When upgrading from Kylin 2.3 ealier version to 2.3 or later, please remove the
“ROLE_” in this setting as this required in the 2.3 earlier version and keep the group
name in original case. And the [Link]-role is deprecated.

Enable LDAP

Set “[Link]=ldap” in conf/[Link], then restart Kylin server.

Enable SSO authentication

From v1.5, Kylin provides SSO with SAML. The implementation is based on Spring
Security SAML Extension. You can read this reference to get an overall understand.

Before trying this, you should have successfully enabled LDAP and managed users with it,
as SSO server may only do authentication, Kylin need search LDAP to get the user’s detail
information.

Generate IDP metadata xml

Contact your IDP (ID provider), asking to generate the SSO metadata file; Usually you
need provide three piece of info:

1. Partner entity ID, which is an unique ID of your app, e.g,: [Link]

name/kylin/saml/metadata
2. App callback endpoint, to which the SAML assertion be posted, it need be: [Link]
name/kylin/saml/SSO
3. Public certificate of Kylin server, the SSO server will encrypt the message with it.

Generate JKS keystore for Kylin

As Kylin need send encrypted message (signed with Kylin’s private key) to SSO server, a
keystore (JKS) need be provided. There are a couple ways to generate the keystore, below
is a sample.

Assume [Link] is the public certificate file, [Link] is the private certificate file; firstly
create a PKCS#12 file with openssl, then convert it to JKS with keytool:

$ openssl pkcs12 -export -in [Link] -inkey [Link] -out kylin.p12

Enter Export Password: <export_pwd>
Verifying - Enter Export Password: <export_pwd>

$ keytool -importkeystore -srckeystore kylin.p12 -srcstoretype PKCS12

-srcstorepass <export_pwd> -alias 1 -destkeystore [Link]
-destalias kylin -destkeypass changeit
Enter destination keystore password: changeit
Re-enter new password: changeit

It will put the keys to “[Link]” with alias “kylin”;

Enable Higher Ciphers

Make sure your environment is ready to handle higher level crypto keys, you may need to
download Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files,
copy local_policy.jar and US_export_policy.jar to $JAVA_HOME/jre/lib/security .

Deploy IDP xml file and keystore to Kylin

The IDP metadata and keystore file need be deployed in Kylin web app’s classpath in
$KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes

1. Name the IDP file to sso_metadata.xml and then copy to Kylin’s classpath;
2. Name the keystore as “[Link]” and then copy to Kylin’s classpath;
3. If you use another alias or password, remember to update that [Link]
accordingly:

<!-- Central storage of cryptographic keys -->

<bean id="keyManager"
class="[Link]">
<constructor-arg value="classpath:[Link]"/>
<constructor-arg type="[Link]" value="changeit"/>
<constructor-arg>
<map>
<entry key="kylin" value="changeit"/>
</map>
</constructor-arg>
<constructor-arg type="[Link]" value="kylin"/>
</bean>

Other configurations

In conf/[Link], add the following properties with your server information:

[Link]=[Link]
[Link]=https
[Link]=host-name
[Link]=443
[Link]=/kylin

Please note, Kylin assume in the SAML message there is a “email” attribute representing
the login user, and the name before @ will be used to search LDAP.
Enable SSO

Set “[Link]=saml” in conf/[Link], then restart Kylin server; After

that, type a URL like “/kylin” or “/kylin/cubes” will redirect to SSO for login, and jump
back after be authorized. While login with LDAP is still available, you can type
“/kylin/login” to use original way. The Rest API (/kylin/api/*) still use LDAP + basic
authentication, no impact.

Install Ranger Plugin

Please refer to [Link]

Enable Zookeeper ACL

Edit $KYLIN_HOME/conf/[Link] to add following configuration item:

 Add “[Link]-auth”. It is the configuration item you can specify the

zookeeper authenticated information. Its formats is “scheme:id”. The value of
scheme that the zookeeper supports is “world”, “auth”, “digest”, “ip” or “super”.
The “id” is the authenticated information of the scheme. For example:

[Link]-auth=digest:ADMIN:KYLIN

The scheme equals to “digest”. The id equals to “ADMIN:KYLIN”, which

expresses the “username:password”.

 Add “[Link]-acl”. It is the configuration item you can set access

permission. Its formats is “scheme:id:permissions”. The value of permissions that
the zookeeper supports is “READ”, “WRITE”, “CREATE”, “DELETE” or
“ADMIN”. For example, we configure that everyone has all the permissions:

[Link]-acl=world:anyone:rwcda

The scheme equals to “world”. The id equals to “anyone” and the permissions
equals to “rwcda”.