# Serverless & EDA Immersion Days @ Cochlear

**WORKSHOP LINK**: https://catalog.us-east-1.prod.workshops.aws/join?access-code=335d-015918-b5

**Workshop help/questions**: [e-mail]

# Day 2 SURVEY! Please complete before leaving!

https://amazonmr.au1.qualtrics.com/jfe/form/SV_blbHTKTeWKRB4dU


# Day 2

## Serverless Developer Experience Workshop

Business workflow:

- Agents create contracts and update contracts
- Agents create listings
- Agents request listing to be evaluated (approved)
- Approved approved listings show up in search
- Property has address as PK. Contract is related to property. Listing ties together both.

Q: Can you please share the lab instructions from Day 1?
> A: Sure! See the bottom of this doc [(link)](#Public-Workshop-Links) :slightly_smiling_face:

Q: What is the advantage of using a named queue rather than just using asynchronously from api gateway to lambda?
> A: It's still perfectly possible to just assynchronously call Lambdas from API Gateway. The reasons we went with an SQS function in between Lambda and API GW include:
> - Ease of implementation, including the ability to build in a DLQ for failed messages
> - Ability to batch-process API requests in Lambda: if API calls are made in rapid succession, we can leverage SQS Batching to process them in groups by single Lambda invocations.
>
> Although that doesn't have a large impact on *this* service, it is a good practice overall for larger, more complex systems

# Day 1

## Questions, Challenges, Comments

> Please add questions or comments in the list below

Q: We have systems that are still running NodeJSv6, how do we enforce ownership and maintenance of IT estate?
> A: By defining a **lightweight**, yet powerful Governance framework. It can be as simple as having a rule like "every resource should be tagged with a Owner, that's an actual person - that ultimately responds for that resource. Any non-tagged resource will be deleted".
> Write the rules in such a way that following them becomes **the path of least resistence**. Make them such that no resource, component, system can be *orphaned* - anything must belong to *someone*...
> 

Q: What can we add to AWS Lambda Layers to help improve performance?

Q: Is there any networking optimisation that can help expedite routing between various service?
> A: In a Serverless context, the network needs to be thought of as *abstracted away*, i.e. not a 'component' to be optimised. Think of the network - in this context - as any other utility in your house. You don't optimise it's delivery, you simply consume it.
>
> That said, if **in a specific situation** your **application requirements** demand more performance from the network that can be delivered, there are techniques one can leverage for better direct network throughput - **at the expense of abstraction** -> e.g. EC2 instances in a Placement Group.

Q: Can we use EDA for applications that are Synchronous in nature? Is there a way to handle this dependency?
> A: EDAs are by their very nature assynchronous. 99.9% of the time, synchronicity can be **engineered away** of an application by good architecture and some *lateral thinking*.
>
> *Personally* I see very few situations where synchronicity is an **absolute hard requirement**. One example would be real-time systems, hardware controllers, etc. But those are definitely the exception, not the rule...

Q: When migrating an on-prem application to cloud, would it be better to do a lift and shift first and then think of re-platform to serverless architecture?
> A: It really depends on a *number* of factors, most important of which include:
> - What is the impact of said application to your business & customers?
> - What is the cost (direct & opportunity) of modernising VS the risk of the app stop working
>
> Basically, what to do with legacy applications **needs** to be evaluated on a case-by-case basis, but there are well-defined strategies and '*frameworks*' that we can help you leverage to simplify this analysis.

Q: Culture of decentralized sounds great. But wouldnt it create an environment of silo when the teams is empowered to decide what language to write or what technology stack to use? An organization will end up 1000s of technologies to pay/license/support.. how do you manage this?
> A: In my experience the degree of *technology sprawl* that *actually* occurs in a de-centralised structure is never this large, for a few reasons:
> - There's not that many *valid* permutations of technologies (even when we consider the many JavaScript frameworks out in the wild :)
> - People often tend to leverage - where it makes sense - technologies already implemented/proven by their peers. We can even **supercharge** that by **fostering a culture of cross-polination**[1] across our teams.
>
> Finally, like most things in IT, it's a trade-off. I propose that the gains of a decentralised structure - agility, independence, scalability, ... - far outweight the potential drawbacks - e.g. tech sprawl - if managed by some **simple** governance rules.

Q: In EDA is the contract between the producer and the consumer?
> A: I guess you could say that... Technically, the contract is **published by the Producer**, who guarantees that events **emitted** will respect the contract, and **leveraged by the Consumers**, who depend on the **guarantee** above to build their own systems and have a **stable** (or at least well-managed) interface with the outside world.

Q: How can we protect sensitive info in Event if any?
> A: **Great question!!** There are a number of techniques we can leverage to protect sensitive information in Events. Which one to use will depend on architecture, organisational and technology characteristics of *your* environment.
>
> One technique is to published *sensitive* data in the events, but encrypted with a key that the producer then controls access to - i.e., only services authorised to use the key will be able to decrypt the sensitive data.
>
> Another technique would be to *not publish* sensitive data in the events, instead 'hiding' it behind an API, that needs to be accessed by the Consumers. Access control is then managed at the API layer. **Yes, this breaks the EDA dependency** model, but it *could* potentially be a valid approach, dependind on the situation...
>
> Finally, there's something to be said about trying to **architect your way out of the problem**... By this I mean: ask if there's nothing you can do to *change your requirements* so to **not need to expose** sensitive information outside of a Domain... Not always possible, but worth investigating :)

Q: How can we ensure ordering of events? For example, if producer publishes a create event followed by an update event. How can consumers know to follow the same order?
> A: Short answer: *most* of the time we can't do that *directly*. Let me explain:
>
> There are services that guarantee delivery ordering, like SQS FIFO queues, or Kafka Topics (ordering is preserved *within each partition*). That, however, comes at a cost: scalability. Everything is a trade-off. By enforcing ordering **in a distributed system**, we have to sacrifice something (see [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem) ): either *partition tolerance* or *availability*.
>
> [IMHO](https://dictionary.cambridge.org/dictionary/english/imho), the question we should be asking is: `Is there any way we don't need to depend on ordering?`
>
> This is another good example of when we should try to **engineer ourselves out of the problem**...

Q: Does Lambda takes care of guaranteed message delivery as well?
> A: I assume you mean messages delivered *to* Lambda via the Event Sources? If so, the Lambda control plane offers [At Least Once](https://blog.bytebytego.com/p/at-most-once-at-least-once-exactly) delivery. You can check out the details in the [Lambda docs](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html#:~:text=Lambda%20event%20source%20mappings%20process,duplicate%20events%20in%20rare%20situations.).

Q: State Manager / Stream tracker --> Is that part of the event source, like Kafka or Kinesis? Or does Lambda manages the counsumer state?
> A: State Manager is an **internal** component of the Lambda service. It acts as a Kafka/Kinesis **client**, by consuming messages from the stream and invoking Lambda functions with the messages as payload.

Q: If we use custom runtime, is there an option to select the runtime as custom while we deploy. For example, we choose nodejs 18 or python 3.7 as runtime while deployment. How does it change when we use custom runtime within our function?
> A: By using a Custom Runtime you necessarily need to choose the `provided.al2` runtime option, as described [here](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html#runtimes-custom-use:~:text=To%20use%20a%20custom%20runtime%2C%20set%20your%20function%27s%20runtime%20to%20provided.al2)

Q: Are there best practices on what goes in to the handler function and what should be defined outside of it?
> A: Absolutely! The rule of thumb is: keep on the *lambda handler* **ONLY WHAT NEEDS TO RUN ON EVERY INVOCATION**.
> Things that can usually go **outside** include, but are not limited to:
> - Initialise DB connection objects
> - Loading configurations from disk or env vars
> - Load static files from disk/s3
> - The list goes on and on and on
> You can check out some of the best practices [here](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html)

# Comments, Concerns, and literally anything else

*Comment*: Love the decentralised mindset, some challenges we see is inter-team dependencies, and knowledge gap in smaller teams.

Q: Tried this command from Creating contracts and got this error.

$ poetry run pytest tests/unit/
ImportError while loading conftest '/home/ec2-user/environment/unicorn/unicorn_contracts/tests/unit/conftest.py'.
tests/unit/conftest.py:6: in <module>
    from aws_lambda_powertools.utilities.typing import LambdaContext
E   ModuleNotFoundError: No module named 'aws_lambda_powertools'

> A: Please run `poetry add aws-lambda-powertools`, then try again :thumbsup:

Q: I'm getting errors when running npm test after updating the contract

During handling of the above exception, another exception occurred:

self = <tests.integration.test_create_contract_apigw.TestCreateContract testMethod=test_create_contract_valid_payload_1>

    def test_create_contract_valid_payload_1(self):
        prop_number = randint(1, 9999)
        payload = override_payload_number(get_event_payload('create_contract_valid_payload_1'), prop_number)
    
        # Call API to create new Contract
        response = requests.post(f'{self.api_endpoint}contracts', json=payload)
        self.properties.append(payload['property_id'])
    
        self.assertEqual(response.status_code, 200)
        self.assertDictEqual(response.json(), response.json() | {"message": "OK"})
    
        sleep(5)
        try:
            eb_event = next(get_cw_logs_values(self.eb_log_group, payload['property_id']))
        except Exception:
>           raise Exception(f'Unable to get EventBridge Event from CloudWatch Logs group {self.eb_log_group}')
E           Exception: Unable to get EventBridge Event from CloudWatch Logs group arn:aws:logs:ap-southeast-2:975435656176:log-group:/aws/events/local/unicorn.contracts-catchall

tests/integration/test_create_contract_apigw.py:58: Exception

> A: Please check if the EventBridge Pipe is implemented. It is required for the end-to-end integration tests.

# Public Workshop Links

- (Day 1) [Building Event-Driven Architectures in AWS](https://catalog.workshops.aws/building-event-driven-architectures-on-aws)
- (Day 2) [Serverless Developer Experience Workshop](https://catalog.workshops.aws/serverless-developer-experience)

# Further reading, listening, or watching
- Blog: [How does lambda scaling compare to other compute options?](https://www.vladionescu.me/posts/scaling-containers-on-aws-in-2022/)
- Blog: [Lambas automatically pre-warm sometimes...](https://aaronstuyvenberg.com/posts/understanding-proactive-initialization)
- YouTube: [Best practices for advanced serverless developers](https://www.youtube.com/watch?v=PiQ_eZFO2GU)
- YouTube: [Advanced serverless workflow patterns and best practices](https://www.youtube.com/watch?v=o6-7BAUWaqg)
- YouTube: [Building Serverlesspresso](https://www.youtube.com/watch?v=qs0U0LdNkV0)
- YouTube: [Idempotency](https://www.youtube.com/watch?v=Jxugill-rOM)
- YouTube: [Shuffling and Sampling](https://www.youtube.com/watch?v=cZ3JEbp4oO0)
- Blog post: [Building and operating a pretty big storage system called S3](https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html)

# Fun Corner

AI Terminal Assistant: https://www.warp.dev/warp-ai (not AWS endorsed)

[xkcd.com](https://xkcd.com) !!