Every week, we run a meetup for the Investec Programmable Banking community. Developers demo cool projects they've been working on and everyone has the chance to ask questions. If you've been wondering where programmable banking is at, here's your sneak peek!
In this demo, Michael Louwrens discusses how he approached generating multiple clients from an API definition to allow for simple updates. Proto3 allows for defining http calls via options, which means that the grpc-gateway creates a proxy for http, which leads to grpc servers. Part of grpc-gateway is a swagger generation component, which allows for swagger definitions to be generated from the proto. This facilitates the easy creation of new clients using swagger-codegen.
The tech he used includes Proto3, grpc-gateway and Swagger Codegen.
Click here to download Michael’s presentation slide deck.
Repo coming soon!
Transcript of the demo
Michael [20:06]
I wanted to use Protobuf for the Investec OpenAPI because it's a nice way to generate a client. If you don't know me, hi, I'm Michael Wallace Louwrens, and I'm building a client generator because I want to be able to generate multiple APIs for whichever language from a single API definition. It allows us to update it pretty easily and I went with what I know, which is using Protobuf in GRPC gateway. So, Swagger Codegen is a bit of a new one for me.
Michael [20:39]
So how does the system work? I'm using Proto3, which allows you to use and to be exact, you can say google.api.http, and you can then define for what is normally used for GRPC messages, you can define how this would look for an HTTP. You can do with GRPC gateway, and you can generate a server that will then be a proxy for any HTTP calls to translate them to GRPC.
Michael [21:20]
Part of this GRPC gateway is that it provides a Swagger generation component. You can then take this proto spec, and you can generate from that and API spec. As I have a lot more experience with [] proto compared to zero I have with working OpenAPI, I decided to go with the solution. So, you can now easily generate Swagger from the proto, and after playing with Swagger-codegen, it's quite easy to create new clients as well.
Michael [21:52]
I mentioned a bit of the why and then the other side of that is Swagger UI is great for demoing an API. My first question for Investec is, I saw /za/pb/v1 and I do have to wonder if you guys aren't using this kind of solution internally.
Willem [22:12]
What do you mean?
Michael [22:14]
GRPC gateway and using protocol definitions.
Willem [22:18]
Yeah, no. That's just our logical grouping so essentially, and we've got various API products running on that same platform with the servers. We've got the UK open banking stuff running there. We are busy with some other secret integrations as well, so that is generally just our structure that we use, and essentially, that also delegates, how traffic gets routed, whether it gets outed into the SA stack or the UK stack and the rest. Hope that it makes sense.
Willem [23:05]
I just wanted to apologise for interrupting you. Did you have any issues with cards or pre-flight when executing the Swaggers?
Michael [23:16]
No, I'm going to demo that in a bit, and I will show all the different parts. So, before we get started, I want to go through the challenges.
Michael [23:28]
I wasted much time trying to get the identity call because it's a bit of a separate service and it just didn't make sense to try doing it in the protofile, but I was very, very stubborn and tried to. If there is somebody who wants to try an alternative to this actually to build it using OpenAPI 3.1 and YAML form, that's one of the alternatives that I might want to try before going and bringing the slides.
Michael [23:56]
At the moment, cURL doesn't work on the Investec API, I can add a browse here, which is what I did testing that cURL is being generated by Swagger, but yeah, cURL doesn't work right now.
Michael [24:15]
And then just a final note, the docs say post everywhere when it should be saying get. Just something I've noticed.
Willem [24:27]
Okay. There's a hard probability for that as the documentation in its current form was crafted during meetings and an hour before going live.
Michael [24:38]
Makes sense.
Willem [24:42 ]
Sorry, again to interrupt you. We are working on just getting our documentation a bit more substantial and just updating that with the current version that's there. Essentially, it is not the right place where it's published, it needs to be in our developer portal and all the rest, so all of that is incoming as well.
Michael [25:10]
Awesome. So, let me quit out here. Let's see where to start. Let me show off what the proto file looks like.
Michael [25:24]
And just this moment, so those of you that haven't seen it we've got, Okay, first, can everyone read the text? Not when I'm busy fiddling with it, I guess.
Ben [25:41]
I can read it. Yes. Yeah, it's good for me.
Michael [25:46]
Awesome. So the parts that aren't so fun, at least with this is, we started here with just the options that go and say, for developer purposes, what the license is, where to find documentation, what schemes the host base are and what security is being used as a default across the documents.
Michael [26:09]
Then you start getting into the individual messages, and that should start looking a bit familiar with the response of the data because it's nice in JSON, I can go and model it here in the proto.
Michael [26:22]
So, we've got "get accounts", we've got "get transactions" with an example of how I can have some default values here. I left the default values blank because right now I am using my account information, but at some point, in time, I will put some example dates and accounts in here. This is just the same information we have seen from the documentation just been put in, and this is what the structure is.
Michael [26:47]
So, we get through all the different data structures, and you can get into the surface, and this short set of roundabouts 40-50 lines is defining all three of these calls. So I am giving a bit of a proto introduction, I do realize that, but you can see here we've got a service account, but then obviously get accounts, which currently is empty because there is no input to it.
Michael [27:16]
It returns a get account response message that I defined above. I say, using the Google API, the HTTP option, do a get call on the path for slash accounts then we just get more information on the summary of the call, a description of the call, and where to find the docs.
Michael [27:37]
When you go and run, I've got a nice little make file here that does all of the general work for me, which is just what it used to generate the actual Swagger and then I just quickly pull the specific version of the Swagger UI and add the override the store actually to point to my Swagger file.
Michael [28:03]
So, to go and see this slide, and let me not make Willem angry and first request a new bearer token.
Willem [28:14]
I don't get angry, just frustrated.
Michael [28:19]
One of the annoying things with OpenAPI [] and then we've got the Swagger spec, and a list of accounts. You can see my accounts over there. You can see the documentation as usual and then get all transactions with the from date to to date.
Michael [28:43]
And that's now the Swagger portion, and if I shut it off, it goes and gives you a cURL request that you can test with, but without adding a header, it's not going to work.
Michael [28:55]
You can go in and see at the bottom here, the data models how everything will look. That's the Swagger side. It's nice to have the Swagger spec, but I figured I should at least have an example client to show off, so I quickly rushed and built a small Python application.
Michael [29:20]
I can now say import Swagger client API and then add a little auth module here that does my login for me, this is now just creating the client, you need to create a configuration object to it, but prefix authorization is the bearer and then provide the function that I created earlier to go and get the key. Then you generate the clients, generate the instance from that client, and then I fetch the information from get accounts.
Michael [29:52]
To run this, you wait a bit, and there it is. If I wanted to change this from get accounts, I can go in and say, accounts, get account balance and let's use the empty account. Okay, so as you can see I haven't gone and done too much with the Python one, and I haven't given my inputs there, but this is now me changing the call. It's just a normal function call as you would normally do. The actual call to generate this is simply this line here. I just extracted the swagger codegen generate of the Swagger file, the language to use an output directory. So, for the final bit seeing, as I saw, somebody picked up the RUST client, I went and generated it with the RUST client as well, which you can see on the side.
Michael [31:16]
This is RUST code, it is usually difficult to read, but I can go and generate many different clients quite easily all from the same Swagger specs. I think, Ben, now would be the time to bring up the poll.
Michael [31:37]
So for next steps, then I'd like to try the OpenAPI or 3.1 as an alternative, I will be asking feedback from the community to go and see which is the preferred way because I'd like to go and share this and I'd like to have a makefile with four different clients and some functional examples.
Michael [31:56]
If this starts getting a bit more traction, then actually publish this to different clients to repositories like NPM and pip. Then some Swagger Codegen flags might be able just to read some of the defaults that come out there by default and say Swagger developers, etc.
Michael [32:16]
So the poll presented here's just a list of the languages that I should have a rough chance of actually being able to include an example client in but I will be able to at least generate a client for so please feel free to go and answer what you'd like to see another example client in.
Ben [32:44]
Oh, I wanted to know, are you gonna be doing this in the future or are we going to be doing this live?
Michael [32:51]
Which one?
Ben [32:54]
Like generating one of these clients.
Michael [32:56]
Well, if you let me do that, and that's a client-generated, obviously you still need to figure out what you are trying to do with it, but that's a new client. We can see the API for accounts here. I haven't done the go one, so I need to go and hunt where the actual function accounts, get account transactions, accounts, get accounts. All this has been generated from the base OpenAPI file.
Ben [33:55]
That's awesome.
Willem [33:59]
Can I just maybe also ask something that came to mind while you were talking?
Willem [34:04]
Somebody in the community tried to consume the API using a Java HTTP client, and then that got blocked. We investigated that. It's not the OpenAPI that's blocking that user agent. It's the underlying channel API that's blocking it and is blocking user agents from PHP, Java, and the rest.
Willem [34:29]
We've got a change going in, and we are coming out of a change freeze on the 1st which means our next deployment slot is Tuesday, the eighth. We're going to do the latest release of our OpenAPI that will allow you to consume the OpenAPI using Java. I saw PHP, and the rest is also blocked currently on our channel API.
Willem [35:02]
I don't know if you ran into any issues in that regard while testing this.
Michael [35:08]
As I mentioned earlier, cURL was blocked. Yeah, it was useless.
Willem [35:13]
Yeah.
Willem [35:14]
It's just a user agent that's been blocked not by the OpenAPI, but the underlying channel API that it proxies the request to. So, we'll fix that on the eighth.
Michael [35:25]
Awesome. Yeah, that's it for me. Any questions?
Willem [35:34]
I see JavaScript there as a client as well, that's not preferred, and the main reason for that is because you're gonna be persisting your AWS credentials, client-side. That's just something to keep in mind. You want to persist your OAuth credential server-side, have your backend API whatever, consume the OpenAPI and then have a separate security protocol between client and your API.
Adam [36:12]
You have discounted the existence of Node js.
Michael [36:15]
I was thinking the same thing.
Willem [36:18]
Okay.
Michael [36:21]
Node for server-side JavaScript in that sense. I got it. Awesome. No other questions?
Ben [36:38]
That's quite slick, Michael. What do you think you want to see people do with it? There's another question. Can we add Adams stuff to this and have those like a community API on top of the investigative API? Adam, does that make any sense what I just said?
Adam [36:58]
What's your intention?
Ben [37:02]
Could we add some of the systems that you built into the Swagger docs and be able to generate clients on top of your thing?
Ben [37:11]
So, as far as I understand you effectively made API into the programmable card in some ways.
Adam [37:17]
It's kind of not directly related. It's just a forwarding mechanism. So, it's separate. I would have to treat I guess we would have to treat putting a Swagger interface for it together separately.
Ben [37:38]
Willem, do you guys still need to publish your own Swagger docs?
Willem [37:53]
We are gonna publish the accounts and transactions in balance, ZA private bank API products into our developer's portal. It's going to serve next to the UK open banking API products. If you look at developer.investec.com that is, for now, a focus on the UK open banking API products is going to look similar to what you see there.
Willem [38:29]
So, we’re like splitting it out, or we are building it out now bringing in the ZA jurisdiction.
Michael [38:44]
There won't be a separate Swagger file that we'll be able to import for generating clients will there?
Adam [38:53]
I'm not sure. Yeah, it's a good question.
Willem [38:59]
Look, I can provide you with the YAML because we must produce it in any case before we can publish our API products onto the developer portal. So, we can provide you guys with the YAML.
Michael [39:17]
Yeah. Because if we can get the YAML then it's more about building the system to go and poll back for updates once in a while, and then go and say, hey, it's time just to go and do a rebuild.
Willem [39:30]
Now that you mention it, I think a useful feature would be to have the ability to download it from the developer portal.
Michael [39:39]
Yes. Oh, yes.
Willem [39:42]
I'll mention that to the guys. I think that will be helpful.
Michael [39:49]
Awesome. Thanks, everyone, for listening.
Ben [39:53]
Thanks, Michael. That's very cool. Is it available online now?
Michael [40:00]
No, this is not available online. I wanted to do the OpenAPI rule spec investigation, instead of me doing it with proto’s intermediaries.
Michael Louwrens is a Product Owner who loves reading, gaming and tech. He’s currently working at Entersekt as a Product Owner for the platform. In his spare time, he likes researching exciting new technologies or new ways of using existing architecture – frequent HN and reddit r/programming browser!
Get involved in the Programmable Banking Community
If you’re not part of our community yet, sign up and join the fun. You can also see more of the demos from our meetups on our YouTube channel here.
For those of you in the community, check out our GitLab to see more of the awesome projects members of our community are working on. You can also sign up for challenges, where you can help find solutions for real life problems.
For more information, get in touch!