Try   HackMD

Logs for AWS S3 on outposts discussion

Adopt s3-outposts during signing

The additional error is not in hivemetastore but in Trino, but both service is add the jar file to adopt s3-outposts signing, I need aws to help me to see the cloudtrail information to see what request is not support

The error log for trino is here

The logs in hivemetastore side is here

Use AWS v4

After adjusting the sign algorithm to v4, we still have 400 BadRequest response.

2023-05-08T01:03:01,293 DEBUG [pool-6-thread-16] http.headers: http-outgoing-1 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAYR27EH6MARMG5SZK/20230508/ap-northeast-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;referer;user-agent;x-amz-content-sha256;x-amz-date, Signature=f53897cc679d33dd9055d1f3a7a7124b3658c1bcd1d97fc76dca74d5a842876a

logs

Head object in hivemetastore

Log 
2023-05-04T08:23:32,495 DEBUG [pool-6-thread-199] s3a.S3AFileSystem: Getting path status for s3a://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3/test/aaa  (test/aaa); needEmptyDirectory=false
2023-05-04T08:23:32,495 DEBUG [pool-6-thread-199] s3a.S3AFileSystem: S3GetFileStatus s3a://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3/test/aaa
2023-05-04T08:23:32,495 DEBUG [pool-6-thread-199] s3a.S3AFileSystem: HEAD test/aaa with change tracker null
2023-05-04T08:23:32,495 DEBUG [pool-6-thread-199] impl.LoggingAuditor: [224] eb838063-5a50-4c49-a852-2a531f53257e-00000008 Executing op_get_file_status with {action_http_head_request 'test/aaa' size=0, mutating=false}; https://audit.example.org/hadoop/1/op_get_file_status/eb838063-5a50-4c49-a852-2a531f53257e-00000008/?op=op_get_file_status&p1=test/aaa&pr=root&ps=348e37ce-8c97-447d-a338-df9044dd738c&id=eb838063-5a50-4c49-a852-2a531f53257e-00000008&t0=224&fs=eb838063-5a50-4c49-a852-2a531f53257e&t1=224&ts=1683188612493
2023-05-04T08:23:32,496 DEBUG [pool-6-thread-199] amazonaws.request: Sending Request: HEAD https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com /test/aaa
2023-05-04T08:23:32,496 DEBUG [pool-6-thread-199] internal.S3Signer: Calculated string to sign:
"HEAD

application/octet-stream
Thu, 04 May 2023 08:23:32 GMT
/ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3/test/aaa"
2023-05-04T08:23:32,497 DEBUG [pool-6-thread-199] protocol.RequestAddCookies: CookieSpec selected: default
2023-05-04T08:23:32,497 DEBUG [pool-6-thread-199] protocol.RequestAuthCache: Auth cache not set in the context
2023-05-04T08:23:32,497 DEBUG [pool-6-thread-199] conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com:443][total available: 0; route allocated: 0 of 96; total allocated: 0 of 96]
2023-05-04T08:23:32,498 DEBUG [pool-6-thread-199] conn.PoolingHttpClientConnectionManager: Connection leased: [id: 3][route: {s}->https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com:443][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
2023-05-04T08:23:32,498 DEBUG [pool-6-thread-199] execchain.MainClientExec: Opening connection {s}->https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com:443
2023-05-04T08:23:32,539 DEBUG [pool-6-thread-199] conn.DefaultHttpClientConnectionOperator: Connecting to ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com/10.0.200.10:443
2023-05-04T08:23:32,541 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory: Connecting socket to ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com/10.0.200.10:443 with timeout 5000
2023-05-04T08:23:32,542 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory: Enabled protocols: [TLSv1.3, TLSv1.2]
2023-05-04T08:23:32,542 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory: Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2023-05-04T08:23:32,542 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory: Starting handshake
2023-05-04T08:23:32,557 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory: Secure session established
2023-05-04T08:23:32,557 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory:  negotiated protocol: TLSv1.2
2023-05-04T08:23:32,557 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory:  negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2023-05-04T08:23:32,557 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory:  peer principal: CN=*.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com
2023-05-04T08:23:32,557 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory:  peer alternative names: [*.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com]
2023-05-04T08:23:32,557 DEBUG [pool-6-thread-199] ssl.SSLConnectionSocketFactory:  issuer principal: CN=Amazon, OU=Server CA 1B, O=Amazon, C=US
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] conn.DefaultHttpClientConnectionOperator: Connection established 10.0.201.237:53838<->10.0.200.10:443
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] conn.DefaultManagedHttpClientConnection: http-outgoing-3: set socket timeout to 200000
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] execchain.MainClientExec: Executing request HEAD /test/aaa HTTP/1.1
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] execchain.MainClientExec: Proxy auth state: UNCHALLENGED
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> HEAD /test/aaa HTTP/1.1
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> Host: ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> amz-sdk-invocation-id: ab2bb9c7-65d7-d304-6e0a-0ebf7afd8da8
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> amz-sdk-request: ttl=20230504T082652Z;attempt=1;max=21
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> amz-sdk-retry: 0/0/500
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> Authorization: AWS AKIAYR27EH6MARMG5SZK:4a494Ljmn5ObJhc8WbUecqvKyyc=
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> Content-Type: application/octet-stream
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> Date: Thu, 04 May 2023 08:23:32 GMT
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> Referer: https://audit.example.org/hadoop/1/op_get_file_status/eb838063-5a50-4c49-a852-2a531f53257e-00000008/?op=op_get_file_status&p1=test/aaa&pr=root&ps=348e37ce-8c97-447d-a338-df9044dd738c&id=eb838063-5a50-4c49-a852-2a531f53257e-00000008&t0=224&fs=eb838063-5a50-4c49-a852-2a531f53257e&t1=224&ts=1683188612493
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> User-Agent: Hadoop 3.3.5, aws-sdk-java/1.12.454 Linux/5.10.178-162.673.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.19+7 java/11.0.19 kotlin/1.4.10 vendor/Eclipse_Adoptium cfg/retry-mode/legacy
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 >> Connection: Keep-Alive
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "HEAD /test/aaa HTTP/1.1[\r][\n]"
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "Host: ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com[\r][\n]"
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "amz-sdk-invocation-id: ab2bb9c7-65d7-d304-6e0a-0ebf7afd8da8[\r][\n]"
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "amz-sdk-request: ttl=20230504T082652Z;attempt=1;max=21[\r][\n]"
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "Authorization: AWS AKIAYR27EH6MARMG5SZK:4a494Ljmn5ObJhc8WbUecqvKyyc=[\r][\n]"
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "Content-Type: application/octet-stream[\r][\n]"
2023-05-04T08:23:32,558 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "Date: Thu, 04 May 2023 08:23:32 GMT[\r][\n]"
2023-05-04T08:23:32,559 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "Referer: https://audit.example.org/hadoop/1/op_get_file_status/eb838063-5a50-4c49-a852-2a531f53257e-00000008/?op=op_get_file_status&p1=test/aaa&pr=root&ps=348e37ce-8c97-447d-a338-df9044dd738c&id=eb838063-5a50-4c49-a852-2a531f53257e-00000008&t0=224&fs=eb838063-5a50-4c49-a852-2a531f53257e&t1=224&ts=1683188612493[\r][\n]"
2023-05-04T08:23:32,559 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "User-Agent: Hadoop 3.3.5, aws-sdk-java/1.12.454 Linux/5.10.178-162.673.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.19+7 java/11.0.19 kotlin/1.4.10 vendor/Eclipse_Adoptium cfg/retry-mode/legacy[\r][\n]"
2023-05-04T08:23:32,559 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "Connection: Keep-Alive[\r][\n]"
2023-05-04T08:23:32,559 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 >> "[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 << "HTTP/1.1 400 Bad Request[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 << "Content-Type: application/xml[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 << "Server: AmazonS3[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 << "x-amz-id-2: tp4ydDqHMY9w6+VuYmNsqwcoqM+lCgzqbWYbwFmnionZISkwNpNV/ZxQsPaRbTrzk0xBvmpALc56yYqU+ax83kN+ReSKYJQ0[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 << "x-amz-request-id: 696441366018D5FD[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 << "Date: Thu, 04 May 2023 08:23:32 GMT[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 << "Content-Length: 314[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.wire: http-outgoing-3 << "[\r][\n]"
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 << HTTP/1.1 400 Bad Request
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 << Content-Type: application/xml
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 << Server: AmazonS3
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 << x-amz-id-2: tp4ydDqHMY9w6+VuYmNsqwcoqM+lCgzqbWYbwFmnionZISkwNpNV/ZxQsPaRbTrzk0xBvmpALc56yYqU+ax83kN+ReSKYJQ0
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 << x-amz-request-id: 696441366018D5FD
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 << Date: Thu, 04 May 2023 08:23:32 GMT
2023-05-04T08:23:32,560 DEBUG [pool-6-thread-199] http.headers: http-outgoing-3 << Content-Length: 314

Head object in aws s3api

Log 
2023-05-04 08:14:25,459 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com
2023-05-04 08:14:25,459 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2023-05-04 08:14:25,459 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'ap-northeast-1', 'signing_name': 's3-outposts', 'disableDoubleEncoding': True}
2023-05-04 08:14:25,459 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.HeadObject: calling handler <function base64_decode_input_blobs at 0x7f2ae74ce980>
2023-05-04 08:14:25,459 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7f2ae98c82c0>
2023-05-04 08:14:25,459 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7f2ae98c8220>
2023-05-04 08:14:25,459 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f2ae98ca2a0>
2023-05-04 08:14:25,459 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f2ae631b690>>
2023-05-04 08:14:25,459 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7f2ae98c8040>
2023-05-04 08:14:25,460 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7f2ae98c85e0>
2023-05-04 08:14:25,460 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function inject_api_version_header_if_needed at 0x7f2ae98c9b20>
2023-05-04 08:14:25,460 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=HeadObject) with params: {'url_path': '/test/aaa', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'aws-cli/2.11.17 Python/3.11.3 Linux/5.10.178-162.673.amzn2.x86_64 docker/x86_64.amzn.2 prompt/off command/s3api.head-object'}, 'body': b'', 'auth_path': '/ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3/test/aaa', 'url': 'https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com/test/aaa', 'context': {'client_region': 'ap-northeast-1', 'client_config': <botocore.config.Config object at 0x7f2ae631ab50>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'ap-northeast-1', 'signing_name': 's3-outposts', 'disableDoubleEncoding': True}, 's3_redirect': {'redirected': False, 'bucket': 'ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3', 'params': {'Bucket': 'ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3', 'Key': 'test/aaa'}}}}
2023-05-04 08:14:25,460 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f2ae6703c50>>
2023-05-04 08:14:25,460 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7f2ae98abec0>
2023-05-04 08:14:25,460 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <function remove_arn_from_signing_path at 0x7f2ae98ca3e0>
2023-05-04 08:14:25,460 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2023-05-04 08:14:25,461 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
HEAD
/test/aaa

host:ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230504T081425Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-05-04 08:14:25,461 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20230504T081425Z
20230504/ap-northeast-1/s3-outposts/aws4_request
51a35d86b119ef571b0f015648538b5d9c15712c124037280f8dd5272fdae609
2023-05-04 08:14:25,461 - MainThread - botocore.auth - DEBUG - Signature:
d06e0f273af14bb2f22ed981fd545ad3dbb793800564edcdd69a651d0abdcf4e
2023-05-04 08:14:25,461 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com/test/aaa, headers={'User-Agent': b'aws-cli/2.11.17 Python/3.11.3 Linux/5.10.178-162.673.amzn2.x86_64 docker/x86_64.amzn.2 prompt/off command/s3api.head-object', 'X-Amz-Date': b'20230504T081425Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAYR27EH6MARMG5SZK/20230504/ap-northeast-1/s3-outposts/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d06e0f273af14bb2f22ed981fd545ad3dbb793800564edcdd69a651d0abdcf4e'}>
2023-05-04 08:14:25,462 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.11.17/dist/awscli/botocore/cacert.pem
2023-05-04 08:14:25,462 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com:443
2023-05-04 08:14:26,486 - MainThread - urllib3.connectionpool - DEBUG - https://ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com:443 "HEAD /test/aaa HTTP/1.1" 200 0
2023-05-04 08:14:26,486 - MainThread - botocore.parsers - DEBUG - Response headers: {'Accept-Ranges': 'bytes', 'Content-Length': '0', 'Content-Type': 'application/octet-stream', 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'Last-Modified': 'Thu, 04 May 2023 08:14:16 GMT', 'Server': 'AmazonS3', 'x-amz-id-2': 'XxhTnfqyE2AtOWESsBwFaDkK5uKpmkbpT+jhSAwXGmwGpNvJowJ5hOqPsJFRgvTIo2P5chckfCy7uiiAUfRDPouYEtDohof9', 'x-amz-request-id': 'E10914E432D8FDC7', 'x-amz-storage-class': 'OUTPOSTS', 'Date': 'Thu, 04 May 2023 08:14:26 GMT'}
2023-05-04 08:14:26,486 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2023-05-04 08:14:26,487 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f2ae633a550>>
2023-05-04 08:14:26,488 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-05-04 08:14:26,488 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f2ae631b690>>
2023-05-04 08:14:26,488 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x7f2ae764b420>
2023-05-04 08:14:26,488 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f2ae62bc990>>
2023-05-04 08:14:26,488 - MainThread - awscli.formatter - DEBUG - RequestId: E10914E432D8FDC7
{
    "AcceptRanges": "bytes",
    "LastModified": "2023-05-04T08:14:16+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "ContentType": "application/octet-stream",
    "Metadata": {},
    "StorageClass": "OUTPOSTS"
}

dig ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10–op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com

Log 
bash-5.1# dig ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com

; <<>> DiG 9.16.39 <<>> ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59951
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 41871cbb5ec21807 (echoed)
;; QUESTION SECTION:
;ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com. IN A

;; ANSWER SECTION:
ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com. 30 IN CNAME ipass-datalake-outpost-dev-access-point-588073156504.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com.
ipass-datalake-outpost-dev-access-point-588073156504.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com. 30 IN A 10.0.200.33
ipass-datalake-outpost-dev-access-point-588073156504.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com. 30 IN A 10.0.200.127
ipass-datalake-outpost-dev-access-point-588073156504.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com. 30 IN A 10.0.200.251
ipass-datalake-outpost-dev-access-point-588073156504.op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com. 30 IN A 10.0.200.10

;; Query time: 47 msec
;; SERVER: 172.20.0.10#53(172.20.0.10)
;; WHEN: Thu May 04 08:27:45 UTC 2023
;; MSG SIZE  rcvd: 939

dig op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com

Log 
bash-5.1# dig op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com

; <<>> DiG 9.16.39 <<>> op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 64681
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 7dd1173b6a9d6b3a (echoed)
;; QUESTION SECTION:
;op-01427051a3dc18b69.s3-outposts.ap-northeast-1.amazonaws.com. IN A

;; AUTHORITY SECTION:
s3-outposts.ap-northeast-1.amazonaws.com. 30 IN	SOA ns-22.awsdns-02.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 39 msec
;; SERVER: 172.20.0.10#53(172.20.0.10)
;; WHEN: Thu May 04 08:28:01 UTC 2023
;; MSG SIZE  rcvd: 225
Last changed by 
 
楊博丞
0
145

Read more

Untitled

2023-05-24 13:16:46,354 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.11.21 Python/3.11.3 Linux/5.10.178-162.673.amzn2.x86_64 docker/x86_64.amzn.2 2023-05-24 13:16:46,354 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: [&apos;s3api&apos;, &apos;put-object&apos;, &apos;--bucket&apos;, &apos;ipass-datala-o01427051a3dc18b69nmaa8betpfvhubhf2siqapn10--op-s3&apos;, &apos;--key&apos;, &apos;test/annual-enterprise-survey-2021-financial-year-provisional-csv.csv&apos;, &apos;--body&apos;, &apos;annual-enterprise-survey-2021-financial-year-provisional-csv.csv&apos;, &apos;--storage-class&apos;, &apos;STANDARD&apos;, &apos;--debug&apos;] 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7f21c36f2e80> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7f21c37300e0> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class &apos;awscli.customizations.configure.configure.ConfigureCommand&apos;>> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f21c387bce0> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f21c3893920> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7f21c36f93a0> 2023-05-24 13:16:46,366 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7f21c376eac0> 2023-05-24 13:16:46,367 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class &apos;awscli.customizations.devcommands.CLIDevCommand&apos;>>

May 24, 2023
Untitled

2023-05-09T12:22:29,841 DEBUG [pool-6-thread-199] metastore.ObjectStore: 0 entries retrieved using SQL in 0.70832ms 2023-05-09T12:22:29,841 INFO [pool-6-thread-199] metastore.ObjectStore: Dropping database hive.test along with all tables 2023-05-09T12:22:29,841 DEBUG [pool-6-thread-199] metastore.ObjectStore: Open transaction: count = 2, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.dropDatabase(ObjectStore.java:1095) java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 2023-05-09T12:22:29,841 DEBUG [pool-6-thread-199] metastore.ObjectStore: Open transaction: count = 3, isActive = true at: org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:968) org.apache.hadoop.hive.metastore.ObjectStore.dropDatabase(ObjectStore.java:1098) java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

May 9, 2023
Untitled

2023-05-09T09:27:31.317Z DEBUG dispatcher-query-15 io.trino.execution.StageStateMachine Stage 20230509_092723_00005_tcfvh.2 is ABORTED 2023-05-09T09:27:31.319Z DEBUG dispatcher-query-17 io.trino.execution.QueryStateMachine Query 20230509_092723_00005_tcfvh is FAILED 2023-05-09T09:27:31.326Z DEBUG page-buffer-client-callback-4 io.trino.operator.HttpPageBufferClient Request to http://10.0.201.203:8080/v1/task/20230509_092723_00005_tcfvh.0.0.0/results/0/0 failed java.util.concurrent.CancellationException: Task was cancelled. 2023-05-09T09:27:31.333Z DEBUG task-notification-3 io.trino.execution.TaskStateMachine Task 20230509_092723_00005_tcfvh.0.0.0 is ABORTING 2023-05-09T09:27:31.338Z DEBUG page-buffer-client-callback-5 io.trino.operator.HttpPageBufferClient Request to http://10.0.201.217:8080/v1/task/20230509_092723_00005_tcfvh.1.0.0/results/0/0 failed java.util.concurrent.CancellationException: Task was cancelled. 2023-05-09T09:27:31.339Z DEBUG 20230509_092723_00005_tcfvh.0.0.0-0-94 io.trino.execution.executor.TaskExecutor Split 20230509_092723_00005_tcfvh.0.0.0-0 (start = 8.9843102223982E7, wall = 6692 ms, cpu = 0 ms, wait = 1 ms, calls = 3) is finished 2023-05-09T09:27:31.342Z DEBUG 20230509_092723_00005_tcfvh.0.0.0-1-92 io.trino.execution.executor.TaskExecutor Split 20230509_092723_00005_tcfvh.0.0.0-1 (start = 8.9843102800127E7, wall = 6695 ms, cpu = 0 ms, wait = 2 ms, calls = 3) is finished 2023-05-09T09:27:31.347Z DEBUG dispatcher-query-8 io.trino.execution.StageStateMachine Stage 20230509_092723_00005_tcfvh.0 is ABORTED 2023-05-09T09:27:31.350Z DEBUG stage-scheduler io.trino.execution.QueryStateMachine Query 20230509_092723_00005_tcfvh failed io.trino.spi.TrinoException: Error committing write to Hive

May 9, 2023
Untitled

2023-05-08T01:03:01,111 DEBUG [pool-6-thread-15] metrics.PerfLogger: <PERFLOG method=get_all_databases from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> 2023-05-08T01:03:01,111 INFO [pool-6-thread-15] metastore.HiveMetaStore: 15: source:10.0.201.237 get_databases: @hive# 2023-05-08T01:03:01,111 INFO [pool-6-thread-15] HiveMetaStore.audit: ugi=root ip=10.0.201.237 cmd=source:10.0.201.237 get_databases: @hive# 2023-05-08T01:03:01,120 INFO [pool-6-thread-15] metastore.HiveMetaStore: 15: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2023-05-08T01:03:01,120 WARN [pool-6-thread-15] metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored 2023-05-08T01:03:01,120 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding datanucleus.schema.autoCreateAll value null from jpox.properties with false 2023-05-08T01:03:01,120 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding javax.jdo.option.ConnectionDriverName value null from jpox.properties with org.postgresql.Driver 2023-05-08T01:03:01,121 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding datanucleus.connectionPool.maxPoolSize value null from jpox.properties with 10 2023-05-08T01:03:01,121 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding datanucleus.connectionPoolingType value null from jpox.properties with HikariCP 2023-05-08T01:03:01,121 DEBUG [pool-6-thread-15] metastore.ObjectStore: Overriding javax.jdo.option.ConnectionURL value null from jpox.properties with jdbc:postgresql://lighthouse-outpost-dev-hivemetastoredb.cm25znb924dp.ap-northeast-1.rds.amazonaws.com:5432/hivemetastoredb?allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC

May 8, 2023
Read more from 楊博丞
Published on HackMD