• Reading & Writing Hive Tables
    • Reading From Hive
    • Writing To Hive
      • Limitations

    Reading & Writing Hive Tables

    Using the HiveCatalog and Flink’s connector to Hive, Flink can read and write from Hive data as an alternative to Hive’s batch engine. Be sure to follow the instructions to include the correct dependencies in your application.

    • Reading From Hive
    • Writing To Hive
      • Limitations

    Reading From Hive

    Assume Hive contains a single table in its default database, named people that contains several rows.

    1. hive> show databases;
    2. OK
    3. default
    4. Time taken: 0.841 seconds, Fetched: 1 row(s)
    6. hive> show tables;
    7. OK
    8. Time taken: 0.087 seconds
    10. hive> CREATE TABLE mytable(name string, value double);
    11. OK
    12. Time taken: 0.127 seconds
    14. hive> SELECT * FROM mytable;
    15. OK
    16. Tom   4.72
    17. John  8.0
    18. Tom   24.2
    19. Bob   3.14
    20. Bob   4.72
    21. Tom   34.9
    22. Mary  4.79
    23. Tiff  2.72
    24. Bill  4.33
    25. Mary  77.7
    26. Time taken: 0.097 seconds, Fetched: 10 row(s)

    With the data ready your can connect to Hive connect to an existing Hive installation and begin querying.

    1. Flink SQL> show catalogs;
    2. myhive
    3. default_catalog
    5. # ------ Set the current catalog to be 'myhive' catalog if you haven't set it in the yaml file ------
    7. Flink SQL> use catalog myhive;
    9. # ------ See all registered database in catalog 'mytable' ------
    11. Flink SQL> show databases;
    12. default
    14. # ------ See the previously registered table 'mytable' ------
    16. Flink SQL> show tables;
    17. mytable
    19. # ------ The table schema that Flink sees is the same that we created in Hive, two columns - name as string and value as double ------ 
    20. Flink SQL> describe mytable;
    21. root
    22.  |-- name: name
    23.  |-- type: STRING
    24.  |-- name: value
    25.  |-- type: DOUBLE
    28. Flink SQL> SELECT * FROM mytable;
    30.    name      value
    31. __________ __________
    33.     Tom      4.72
    34.     John     8.0
    35.     Tom      24.2
    36.     Bob      3.14
    37.     Bob      4.72
    38.     Tom      34.9
    39.     Mary     4.79
    40.     Tiff     2.72
    41.     Bill     4.33
    42.     Mary     77.7

    Writing To Hive

    Similarly, data can be written into hive using an INSERT INTO clause.

    1. Flink SQL> INSERT INTO mytable (name, value) VALUES ('Tom', 4.72);

    Limitations

    The following is a list of major limitations of the Hive connector. And we’re actively working to close these gaps.

    • INSERT OVERWRITE is not supported.
    • Inserting into partitioned tables is not supported.
    • ACID tables are not supported.
    • Bucketed tables are not supported.
    • Some data types are not supported. See the limitations for details.
    • Only a limited number of table storage formats have been tested, namely text, SequenceFile, ORC, and Parquet.
    • Views are not supported.