1.1. Installation¶

1.1.1.1. Hadoop dependencies¶

Sqoop server needs following environmental variables pointing at Hadoop libraries - $HADOOP_COMMON_HOME, $HADOOP_HDFS_HOME, $HADOOP_MAPRED_HOME and $HADOOP_YARN_HOME. You have to make sure that those variables are defined and pointing to a valid Hadoop installation. Sqoop server will not start if Hadoop libraries can’t be found.

The Sqoop server uses environment variables to find Hadoop libraries. If the environment variable $HADOOP_HOME is set, Sqoop will look for jars in the following locations: $HADOOP_HOME/share/hadoop/common, $HADOOP_HOME/share/hadoop/hdfs, $HADOOP_HOME/share/hadoop/mapreduce and $HADOOP_HOME/share/hadoop/yarn. You can specify where the Sqoop server should look for the common, hdfs, mapreduce, and yarn jars indepently with the $HADOOP_COMMON_HOME, $HADOOP_HDFS_HOME, $HADOOP_MAPRED_HOME and $HADOOP_YARN_HOME environment variables.

# Export HADOOP_HOME variable
export HADOOP_HOME=/...

# Or alternatively HADOOP_*_HOME variables
export HADOOP_COMMON_HOME=/...
export HADOOP_HDFS_HOME=/...
export HADOOP_MAPRED_HOME=/...
export HADOOP_YARN_HOME=/...

1.1.1.2. Hadoop configuration¶

Sqoop server will need to impersonate users to access HDFS and other resources in or outside of the cluster as the user who started given job rather then user who is running the server. You need to configure Hadoop to explicitly allow this impersonation via so called proxyuser system. You need to create two properties in core-site.xml file - hadoop.proxyuser.$SERVER_USER.hosts and hadoop.proxyuser.$SERVER_USER.groups where $SERVER_USER is the user who will be running Sqoop 2 server. In most scenarios configuring * is sufficient. Please refer to Hadoop documentation for details how to use those properties.

Example fragment that needs to be present in core-site.xml file for case when server is running under sqoop2 user:

<property>
  <name>hadoop.proxyuser.sqoop2.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.sqoop2.groups</name>
  <value>*</value>
</property>

If you’re running Sqoop 2 server under a so called system user (user with ID less then min.user.id - 1000 by default), then YARN will by default refuse to run Sqoop 2 jobs. You will need to add the user name who is running Sqoop 2 server (most likely user sqoop2) to a allowed.system.users property of container-executor.cfg. Please refer to YARN documentation for further details.

Example fragment that needs to be present in container-executor.cfg file for case when server is running under sqoop2 user:

allowed.system.users=sqoop2

1.1.1.3. Third party jars¶

To propagate any third party jars to Sqoop server classpath, create a directory anywhere on the file system and export it’s location in SQOOP_SERVER_EXTRA_LIB variable.

# Create directory for extra jars
mkdir -p /var/lib/sqoop2/

# Copy all your JDBC drivers to this directory
cp mysql-jdbc*.jar /var/lib/sqoop2/
cp postgresql-jdbc*.jar /var/lib/sqoop2/

# And finally export this directory to SQOOP_SERVER_EXTRA_LIB
export SQOOP_SERVER_EXTRA_LIB=/var/lib/sqoop2/

1.1.1.4. Configuring PATH¶

All user and administrator facing shell commands are stored in bin/ directory. It’s recommended to add this directory to your $PATH for easier execution, for example:

PATH=$PATH:`pwd`/bin/

The remainder of the Sqoop 2 documentation assumes that the shell commands are in your $PATH.

1.1.1.5. Configuring Server¶

Server configuration files are stored in conf directory. File sqoop_bootstrap.properties specifies which configuration provider should be used for loading configuration for rest of Sqoop server. Default value PropertiesConfigurationProvider should be sufficient.

Second configuration file called sqoop.properties contains remaining configuration properties that can affect Sqoop server. The configuration file is very well documented, so check if all configuration properties fits your environment. Default or very little tweaking should be sufficient in most common cases.

1.1.1.6. Repository Initialization¶

The metadata repository needs to be initialized before starting Sqoop 2 server for the first time. Use Upgrade tool to initialize the repository:

sqoop2-tool upgrade

You can verify if everything have been configured correctly using Verify tool:

sqoop2-tool verify
...
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly

1.1.1.7. Server Life Cycle¶

After installation and configuration you can start Sqoop server with following command:

sqoop2-server start

You can stop the server using the following command:

sqoop2-server stop

By default Sqoop server daemon use port 12000. You can set org.apache.sqoop.jetty.port in configuration file conf/sqoop.properties to use different port.