楊博丞

@BochengYang

Joined on Feb 16, 2017

  • 2023-05-24 13:16:46,354 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.11.21 Python/3.11.3 Linux/5.10.178-162.673.amzn2.x86_64 docker/x86_64.amzn.2 2023-05-24 13:16:46,354 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3api', 'put-object', '--bucket', 'ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3', '--key', 'test/annual-enterprise-survey-2021-financial-year-provisional-csv.csv', '--body', 'annual-enterprise-survey-2021-financial-year-provisional-csv.csv', '--storage-class', 'STANDARD', '--debug'] 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7f21c36f2e80> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7f21c37300e0> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f21c387bce0> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f21c3893920> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7f21c36f93a0> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7f21c376eac0> 2023-05-24 13:16:46,367 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
     Like  Bookmark
  • Adopt s3-outposts during signing The additional error is not in hivemetastore but in Trino, but both service is add the jar file to adopt s3-outposts signing, I need aws to help me to see the cloudtrail information to see what request is not support The error log for trino is here The logs in hivemetastore side is here Use AWS v4 After adjusting the sign algorithm to v4, we still have 400 BadRequest response.
     Like  Bookmark
  • 2023-05-09T12:22:29,841 DEBUG [pool-6-thread-199] metastore.ObjectStore: 0 entries retrieved using SQL in 0.70832ms 2023-05-09T12:22:29,841 INFO [pool-6-thread-199] metastore.ObjectStore: Dropping database hive.test along with all tables 2023-05-09T12:22:29,841 DEBUG [pool-6-thread-199] metastore.ObjectStore: Open transaction: count = 2, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.dropDatabase(ObjectStore.java:1095) java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 2023-05-09T12:22:29,841 DEBUG [pool-6-thread-199] metastore.ObjectStore: Open transaction: count = 3, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:968) org.apache.hadoop.hive.metastore.ObjectStore.dropDatabase(ObjectStore.java:1098) java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     Like  Bookmark
  • 2023-05-09T09:27:31.317Z DEBUG dispatcher-query-15 io.trino.execution.StageStateMachine Stage 20230509_092723_00005_tcfvh.2 is ABORTED 2023-05-09T09:27:31.319Z DEBUG dispatcher-query-17 io.trino.execution.QueryStateMachine Query 20230509_092723_00005_tcfvh is FAILED 2023-05-09T09:27:31.326Z DEBUG page-buffer-client-callback-4 io.trino.operator.HttpPageBufferClient Request to http://10.0.201.203:8080/v1/task/20230509_092723_00005_tcfvh.0.0.0/results/0/0 failed java.util.concurrent.CancellationException: Task was cancelled. 2023-05-09T09:27:31.333Z DEBUG task-notification-3 io.trino.execution.TaskStateMachine Task 20230509_092723_00005_tcfvh.0.0.0 is ABORTING 2023-05-09T09:27:31.338Z DEBUG page-buffer-client-callback-5 io.trino.operator.HttpPageBufferClient Request to http://10.0.201.217:8080/v1/task/20230509_092723_00005_tcfvh.1.0.0/results/0/0 failed java.util.concurrent.CancellationException: Task was cancelled. 2023-05-09T09:27:31.339Z DEBUG 20230509_092723_00005_tcfvh.0.0.0-0-94 io.trino.execution.executor.TaskExecutor Split 20230509_092723_00005_tcfvh.0.0.0-0 (start = 8.9843102223982E7, wall = 6692 ms, cpu = 0 ms, wait = 1 ms, calls = 3) is finished 2023-05-09T09:27:31.342Z DEBUG 20230509_092723_00005_tcfvh.0.0.0-1-92 io.trino.execution.executor.TaskExecutor Split 20230509_092723_00005_tcfvh.0.0.0-1 (start = 8.9843102800127E7, wall = 6695 ms, cpu = 0 ms, wait = 2 ms, calls = 3) is finished 2023-05-09T09:27:31.347Z DEBUG dispatcher-query-8 io.trino.execution.StageStateMachine Stage 20230509_092723_00005_tcfvh.0 is ABORTED 2023-05-09T09:27:31.350Z DEBUG stage-scheduler io.trino.execution.QueryStateMachine Query 20230509_092723_00005_tcfvh failed io.trino.spi.TrinoException: Error committing write to Hive
     Like  Bookmark
  • 2023-05-08T01:03:01,111 DEBUG [pool-6-thread-15] metrics.PerfLogger: <PERFLOG method=get_all_databases from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> 2023-05-08T01:03:01,111 INFO [pool-6-thread-15] metastore.HiveMetaStore: 15: source:10.0.201.237 get_databases: @hive# 2023-05-08T01:03:01,111 INFO [pool-6-thread-15] HiveMetaStore.audit: ugi=root ip=10.0.201.237 cmd=source:10.0.201.237 get_databases: @hive# 2023-05-08T01:03:01,120 INFO [pool-6-thread-15] metastore.HiveMetaStore: 15: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2023-05-08T01:03:01,120 WARN [pool-6-thread-15] metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored 2023-05-08T01:03:01,120 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding datanucleus.schema.autoCreateAll value null from jpox.properties with false 2023-05-08T01:03:01,120 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding javax.jdo.option.ConnectionDriverName value null from jpox.properties with org.postgresql.Driver 2023-05-08T01:03:01,121 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding datanucleus.connectionPool.maxPoolSize value null from jpox.properties with 10 2023-05-08T01:03:01,121 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding datanucleus.connectionPoolingType value null from jpox.properties with HikariCP 2023-05-08T01:03:01,121 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding javax.jdo.option.ConnectionURL value null from jpox.properties with jdbc:postgresql://lighthouse-outpost-dev-hivemetastoredb.cm25znb924dp.ap-northeast-1.rds.amazonaws.com:5432/hivemetastoredb?allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC
     Like  Bookmark
  • This article is used to describe the issue we've encountered during the integration of hivemetastore, trino, and s3 on outposts CASE1: Specified s3 endpoint to https://s3-outposts.ap-northeast-1.amazonaws.com In this case, when we set the s3 endpoint to https://s3-outposts.ap-northeast-1.amazonaws.com, we can see the debug message generated from hivemetastore from another note In conclusion of this debug and error message, it looks like this is an unsupported operation CASE2: Specified s3 endpoint to https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com When we use another endpoint that AWS metioned, the error message looks like sending a wrong parameter to the endpoint, the detail of the debug and error message in this note
     Like  Bookmark
  • 2023-05-04T03:59:21,018 DEBUG [pool-6-thread-198] metrics.PerfLogger: </PERFLOG method=create_database start=1683172451202 end=1683172761018 duration=309816 from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=196 retryCount=-1 error=true> 2023-05-04T03:59:21,018 INFO [pool-6-thread-198] metastore.HiveMetaStore: 196: Cleaning up thread local RawStore... 2023-05-04T03:59:21,018 INFO [pool-6-thread-198] HiveMetaStore.audit: ugi=root ip=10.0.200.131 cmd=Cleaning up thread local RawStore... 2023-05-04T03:59:21,018 DEBUG [pool-6-thread-198] metastore.ObjectStore: RawStore: org.apache.hadoop.hive.metastore.ObjectStore@612e4728, with PersistenceManager: org.datanucleus.api.jdo.JDOPersistenceManager@7c7fcf53 will be shutdown 2023-05-04T03:59:21,018 INFO [pool-6-thread-198] metastore.HiveMetaStore: 196: Done cleaning up thread local RawStore 2023-05-04T03:59:21,018 INFO [pool-6-thread-198] HiveMetaStore.audit: ugi=root ip=10.0.200.131 cmd=Done cleaning up thread local RawStore 2023-05-04T03:59:22,939 INFO [pool-6-thread-198] metastore.HiveMetaStore: 196: Done cleaning up thread local RawStore 2023-05-04T03:59:22,939 INFO [pool-6-thread-198] HiveMetaStore.audit: ugi=root ip=10.0.200.168 cmd=Done cleaning up thread local RawStore 2023-05-04T03:59:22,939 INFO [pool-6-thread-199] metastore.HiveMetaStore: 195: Done cleaning up thread local RawStore 2023-05-04T03:59:22,939 INFO [pool-6-thread-199] HiveMetaStore.audit: ugi=root ip=10.0.200.168 cmd=Done cleaning up thread local RawStore
     Like  Bookmark
  • 2023-05-02T02:16:49,746 DEBUG [pool-6-thread-19] s3a.S3AFileSystem: Getting path status for s3a://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3/test (test); needEmptyDirectory=false 2023-05-02T02:16:49,746 DEBUG [pool-6-thread-19] s3a.S3AFileSystem: S3GetFileStatus s3a://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3/test 2023-05-02T02:16:49,747 DEBUG [pool-6-thread-19] s3a.S3AFileSystem: HEAD test with change tracker null 2023-05-02T02:16:49,747 DEBUG [pool-6-thread-19] impl.LoggingAuditor: [44] b8c151e2-fe99-4d7c-a14a-62ffd59159ce-00000006 Executing op_get_file_status with {action_http_head_request 'test' size=0, mutating=false}; https://audit.example.org/hadoop/1/op_get_file_status/b8c151e2-fe99-4d7c-a14a-62ffd59159ce-00000006/?op=op_get_file_status&p1=test&pr=root&ps=3163699b-bd34-4b6a-8686-d17e18b10e85&id=b8c151e2-fe99-4d7c-a14a-62ffd59159ce-00000006&t0=44&fs=b8c151e2-fe99-4d7c-a14a-62ffd59159ce&t1=44&ts=1682993809742 2023-05-02T02:16:49,748 DEBUG [pool-6-thread-19] amazonaws.request: Sending Request: HEAD https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com /ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3/test 2023-05-02T02:16:49,749 DEBUG [pool-6-thread-19] internal.S3Signer: Calculated string to sign: "HEAD application/octet-stream Tue, 02 May 2023 02:16:49 GMT
     Like  Bookmark