Session level API caching

[bigkraig]

No description provided.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bigkraig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [bigkraig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Coverage increased (+0.5%) to 28.868% when pulling 6ae8e16 on session-caching into f959dc6 on master.

[M00nF1sh]

Nice work! 👍
Remaining question before putting the merge label 😸 .

1. should we leverage LayeredCache, instead of sync.Map?(however, there will be no changes in this PR)

2. should we directly embed AWS API in our wrapper API interfaces? It's convenient, but we are locked to specific aws go sdk version (unrelated to this PR, but just wondering what should the correct abstraction level in our AWS API wrapper, and whether we should do cache there, since caching on session level is "kind of hacky")

3. should we indeed need caching, once we implemented "only refresh specific ingress when changes happen"?

[bigkraig]

1. That's a good idea. Can backlog it on the caching package.
2. I need to be convinced on having an opinion on this one way or the other.. I'm not really worried about being locked to the AWS SDK version, maybe I should be?
3. We will probably still need it for larger deployments. The API throttles at relatively low rates and is troublesome to track down what is making API calls when you hit the limit. Anything we can do to reduce superfluous queries is good in my book.

[M00nF1sh]

For #2, actually i'm ok with both solution if we adding more integration test cases for the caching(just incase aws break the internal implementation some how).

For #3, i'm afraid our local caching might cause problems. E.g. our ingress controller wants to update securityGroups on nodes, but it overrides changes make by other components/user due to there is new API like addSecurityGroup, and aws doesn't support compareAndSet.

[M00nF1sh]

/lgtm