Skip to content

Commit f80a821

Browse files
CarlDownsOracleCarlDownsOracle
authored
Merge pull request oracle-samples#48 from oracle-samples/carl-downs-batch-1
initial check in of oci-monitoring-metrics-to-datadog-python
2 parents 29c490a + 354e3a3 commit f80a821

File tree

10 files changed

+595
-0
lines changed

10 files changed

+595
-0
lines changed

‎samples/oci-monitoring-metrics-to-datadog-python/.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
.DS_Store
2+
.project
3+
.classpath
4+
.settings
5+
target

‎samples/oci-monitoring-metrics-to-datadog-python/LICENSE.txt

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
Copyright (c) 2022, Oracle and/or its affiliates. All rights reserved.
2+
3+
The Universal Permissive License (UPL), Version 1.0
4+
5+
Subject to the condition set forth below, permission is hereby granted to any person obtaining a copy of this
6+
software, associated documentation and/or data (collectively the "Software"), free of charge and under any
7+
and all copyright rights in the Software, and any and all patent rights owned or freely licensable by each
8+
licensor hereunder covering either (i) the unmodified Software as contributed to or provided by such licensor,
9+
or (ii) the Larger Works (as defined below), to deal in both
10+
11+
(a) the Software, and
12+
13+
(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if one is included with the
14+
Software (each a “Larger Work” to which the Software is contributed by such licensors), without restriction,
15+
including without limitation the rights to copy, create derivative works of, display, perform, and
16+
distribute the Software and make, use, sell, offer for sale, import, export, have made, and have sold
17+
the Software and the Larger Work(s), and to sublicense the foregoing rights on either these or other terms.
18+
19+
This license is subject to the following condition:
20+
21+
The above copyright notice and either this complete permission notice or at a minimum a reference to the
22+
UPL must be included in all copies or substantial portions of the Software.
23+
24+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
25+
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
26+
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
27+
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE
28+
OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎samples/oci-monitoring-metrics-to-datadog-python/README.md

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
# Exporting OCI Monitoring Service Metrics to Datadog
2+
3+
---
4+
5+
## Overview
6+
7+
Let's take a look at bringing Oracle Cloud Infrastructure (OCI)’s rich Metrics resources over to
8+
Datadog to accomplish common goals such DevOps monitoring, application performance monitoring, and so on.
9+
Datadog’s API exposes some unique features. Their API allows you to characterize each metric using tags,
10+
which is essential to aggregating and correlating data as needed for its monitoring, reporting, dashboards, and so on.
11+
12+
Please see the
13+
[companion blog](https://blogs.oracle.com/cloud-infrastructure/post/exporting-oci-monitoring-service-metrics-to-datadog) for more details.
14+
15+
### Prerequisites
16+
17+
If you’re new to Functions, get familiar by running through
18+
the [Quick Start guide on OCI Functions](http://docs.oracle.com/en-us/iaas/Content/Functions/Tasks/functionsquickstartguidestop.htm) before proceeding.
19+
20+
---
21+
## Solution Architecture
22+
23+
![](images/architecture.png)
24+
25+
Here is the basic architecture and flow of data from beginning to end:
26+
27+
* OCI services emit metric data which is captured by the Monitoring service.
28+
* The Monitoring Service feeds metric data events to a Service Connector.
29+
* The Service Connector invokes a Function which transforms the metric data payload to Datadog format and posts the transformed payload to the Datadog REST API.
30+
* Datadog ingests the metrics, building its own aggregations using the provided tagging.
31+
32+
Let's drill down into the OCI Services involved.
33+
34+
---
35+
## Monitoring Service
36+
37+
The [Monitoring Service](https://docs.oracle.com/en-us/iaas/Content/Monitoring/Concepts/monitoringoverview.htm)
38+
receives timestamp-value pairs (aka metric data points) which also carry contextual
39+
dimensions and metadata about the services or applications that emitted them.
40+
41+
---
42+
## Service Connector Hub
43+
44+
The stream of Metric data is event-driven and must be handled on-demand and at scale. The
45+
[Service Connector Hub](https://docs.oracle.com/en-us/iaas/Content/service-connector-hub/overview.htm) does
46+
exactly that. See [Service Connector Hub documentation](https://docs.oracle.com/en-us/iaas/Content/service-connector-hub/overview.htm) for details.
47+
48+
---
49+
## Functions Service
50+
51+
I need to transform between the raw metrics formats and some way to make the Datadog API calls. The
52+
[OCI Functions Service](http://docs.oracle.com/en-us/iaas/Content/Functions/Concepts/functionsoverview.htm) is a
53+
natural fit for the task. Functions integrate nicely with Service Connector Hub as as a target and can scale up
54+
depending on the demand. That lets me focus on writing the logic needed without needing to address how to
55+
deploy and scale it.
56+
57+
---
58+
## Mapping From OCI to DataDog Formats
59+
60+
A key requirement of course is the mapping of OCI to Datadog format. Let's compare the OCI and Datadog
61+
message payload formats, what the mapping needs to accomplish, and see what the resulting transformed message
62+
looks like.
63+
64+
Example OCI Metrics Payload:
65+
66+
{
67+
"namespace": "oci_vcn",
68+
"resourceGroup": null,
69+
"compartmentId": "ocid1.compartment.oc1...",
70+
"name": "VnicFromNetworkBytes",
71+
"dimensions": {
72+
"resourceId": "ocid1.vnic.oc1.phx..."
73+
},
74+
"metadata": {
75+
"displayName": "Bytes from Network",
76+
"unit": "bytes"
77+
},
78+
"datapoints": [
79+
{
80+
"timestamp": 1652196912000,
81+
"value": 5780.0,
82+
"count": 1
83+
}
84+
]
85+
}
86+
87+
Example DataDog Metrics Payload:
88+
89+
{
90+
"series": [
91+
{
92+
"metric": "system.load.1",
93+
"type": 0,
94+
"points": [
95+
{
96+
"timestamp": 1636629071,
97+
'value": 1.1
98+
}
99+
],
100+
"tags": [
101+
"test:ExampleSubmitmetricsreturnsPayloadacceptedresponse"
102+
]
103+
}
104+
]
105+
}
106+
107+
Mapping Behavior:
108+
109+
{
110+
"series": [
111+
{
112+
"metric": "{re-characterized OCI namespace and metric name values}",
113+
"type": {mapped_type_enum},
114+
"points": [
115+
{
116+
"timestamp": {datapoint.timestamp},
117+
"value": {datapoint.value}
118+
}
119+
],
120+
"tags": [
121+
"{metrics tag key1:oci payload value}",
122+
"{metrics tag key2:oci payload_value}"
123+
]
124+
}
125+
]
126+
}
127+
128+
Resulting Output:
129+
130+
{
131+
"series": [
132+
{
133+
"metric": "oci.vcn.vnic.from.network.bytes",
134+
"type": 0,
135+
"points": [
136+
{
137+
"timestamp": 1652196912,
138+
"value": 5780.0
139+
}
140+
],
141+
"tags": [
142+
"name:VnicFromNetworkBytes",
143+
"unit:bytes",
144+
"namespace:oci_vcn",
145+
"displayName:Bytes from Network"
146+
]
147+
}
148+
]
149+
}
150+
151+
---
152+
## Policy Setup
153+
154+
You will need
155+
this [IAM policy](https://docs.oracle.com/en-us/iaas/Content/Functions/Tasks/functionscreatingpolicies.htm#Create_Policies_to_Control_Access_to_Network_and_FunctionRelated_Resources)
156+
to authorize the Service Connector to invoke your Function.
157+
158+
allow any-user to use fn-function in compartment id ocid1.compartment.oc1... where all {request.principal.type=’serviceconnector’, request.principal.compartment.id=’ocid1.compartment.oc1...’}
159+
160+
---
161+
## Service Connector Setup
162+
163+
Now let’s set up a simple service connector instance that takes Monitoring sources and passes them to our Function.
164+
165+
Because your Function requires a VCN, you can use that VCN as the metric source to test against. Let's test
166+
with the `oci_vcn` Monitoring namespace because it will quickly generate a lot of useful events.
167+
168+
Select Monitoring as the source and the Function as the target. Configure your source as the
169+
compartment where the VCN resides and select the Monitoring namespace (`oci_vcn`) that you want to
170+
pick up. Select your Application and the Function within it as the target.
171+
172+
<br />
173+
174+
[<img src="images/sch-setup.png" width="800"/>](image.png)
175+
176+
---
177+
## View Metrics In DataDog
178+
179+
When you have the Service Connector configured, metrics appear in Datadog's Metrics Explorer and notebooks
180+
after a few minutes. The following images show the Metrics Explorer and Notebook user interfaces in
181+
Datadog. Your VCN metrics are displayed.
182+
183+
184+
[<img src="images/datadog1.png" width="800"/>](image.png)
185+
186+
<br />
187+
188+
[<img src="images/datadog2.png" width="800"/>](image.png)
189+
190+
---
191+
## Function Environment
192+
193+
Here are the supported Function parameters:
194+
195+
| Environment Variable | Default | Purpose |
196+
| ------------- |:-------------:| :----- |
197+
| DATADOG_METRICS_API_ENDPOINT | not-configured | REST API endpoint for reaching DataDog ([see docs](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics))|
198+
| DATADOG_API_TOKEN | not-configured | API license token obtained from DataDog |
199+
| METRICS_TAG_KEYS | name, namespace, displayName, resourceDisplayName, unit | OCI Metric Dimensions and metadata to convert to DataDog Metric Tags |
200+
| LOGGING_LEVEL | INFO | Controls function logging outputs. Choices: INFO, WARN, CRITICAL, ERROR, DEBUG |
201+
| ENABLE_TRACING | False | Enables complete exception stack trace logging |
202+
| FORWARD_TO_DATADOG | True | Determines whether messages are forwarded to DataDog |
203+
204+
---
205+
## Conclusion
206+
207+
You now have a low-maintenance, serverless function that can send raw metrics over to DataDog in
208+
near-real time. I encourage you to experiment with the dimensions and metadata tag mappings
209+
to see which combination works best for your use case.
210+
211+
For more information, see the following resources:
212+
213+
- [DataDog Metrics API Reference](https://docs.datadoghq.com/api/latest/metrics/)
214+
- [DataDog Metrics API / Submit Metrics API contract](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics)
215+
216+
---
217+
## **OCI** Related Workshops
218+
219+
LiveLabs is the place to explore Oracle's products and services using workshops designed to
220+
enhance your experience building and deploying applications on the Cloud and On-Premises.
221+
ur library of workshops cover everything from how to provision the world's first autonomous
222+
database to setting up a webserver on our world class OCI Generation 2 infrastructure,
223+
machine learning and much more. Use your existing Oracle Cloud account,
224+
a [Free Tier](https://www.oracle.com/cloud/free/) account or a LiveLabs Cloud Account to build, test,
225+
and deploy applications on Oracle's Cloud.
226+
227+
Visit [LiveLabs](http://bit.ly/golivelabs) now to get started. Workshops are added weekly, please visit frequently for new content.
228+
229+
---
230+
## License
231+
Copyright (c) 2022, Oracle and/or its affiliates. All rights reserved.
232+
Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.

0 commit comments

Comments
 (0)