Development

Fetch EXIF has been upgraded

Reduce query times and traffic, optimize cache logic, and support Docker deployment

2023年4月27日
Fetch EXIF has been upgraded
本文共有835字，预计阅读时间4分钟
I'm learning Node recently and I'm going to write a backend project. Just after reading how to CRUD, I decided to upgrade a small project used by this blog.

Previous Question

If you randomly click on the photography works on this site, you will find that the shooting data will be displayed under each picture.
The function of Fetch EXIF is very simple. It receives a picture URL and returns the shooting information of the picture.
I wrote an article about this feature before: 利用ChatGPT实现一个Node.js API。
But there are some problems with this API basically completed by ChatGPT, at that time I was eager to complete the function without considering the performance. After using it for so long, although I didn't really encounter a performance bottleneck, I always wanted to optimize it.
The previous API had two performance issues.
The first one is the extraction of EXIF and GPS. Previously I split the two queries into two parts, but actually GPS is part of EXIF. Now that there is already a cache, the two parts of the query can actually be combined into one, and different information can be returned separately.
Doing so saves one query per page. (Because a photographic work will only read GPS once, EXIF is included in every picture).
The second is the way of caching. At the beginning, I took ChatGPT's suggestion and stored the query data in memory through node-cache. It stands to reason that even if there are tens of thousands of pictures, that bit of information is enough to store in memory. But after all, it is not a persistent solution. Once the program is restarted, the pictures have to be queried and cached again. If the data can be stored in the database, it should be able to save a lot of traffic for downloading pictures. (There is actually no difference in response speed)

Refactor

So I refactored the project. First define the Schema of MongoDB:
1const mongoose = require('mongoose');
2
3const imageSchema = new mongoose.Schema({
4  url: { type: String, required: true, unique: true },
5  exif: {
6    Maker: { type: String, default: 'unknown' },
7    Model: { type: String, default: 'unknown' },
8    ExposureTime: { type: String, default: 'unknown' },
9    FNumber: { type: String, default: 'unknown' },
10    iso: { type: String, default: 'unknown' },
11    FocalLength: { type: String, default: 'unknown' },
12    LensModel: { type: String, default: 'unknown' }
13  },
14  gps: {
15    latitude: { type: Number, required: true, default: 0 },
16    longitude: { type: Number, required: true, default: 0 }
17  },
18  createdAt: { type: Date, expires: '30d', default: Date.now }
19})
20
21module.exports = {
22  Raw: mongoose.model('Raw', imageSchema)
23}
Then refactor the function to obtain shooting information, and query and store EXIF and GPS information at one time:
1const exifr = require("exifr");
2const { Raw } = require("./database");
3
4//将快门数据转换成更容易理解的格式
5function formatShutterTime(shutterTime) {
6  if (!shutterTime) return "0";
7  const time = parseFloat(shutterTime);
8  if (time >= 1) {
9    return time.toFixed(2);
10  }
11  const fraction = Math.round(1 / time);
12  return `1/${fraction}`;
13}
14
15//将axios改为直接使用fetch()，查询到数据后存入数据库
16async function getRaw(url) {
17  try {
18    const response = await fetch(url);
19    const data = new Uint8Array(await response.arrayBuffer());
20    const rawData = await exifr.parse(data);
21    const image = new Raw({
22      url,
23      exif: {
24        Maker: rawData.Make || "unknown",
25        Model: rawData.Model || "unknown",
26        ExposureTime: formatShutterTime(rawData.ExposureTime),
27        FNumber: rawData.FNumber || "unknown",
28        iso: rawData.ISO || "unknown",
29        FocalLength: rawData.FocalLength || "unknown",
30        LensModel: rawData.LensModel || "unknown",
31      },
32      gps: {
33        latitude: rawData.latitude,
34        longitude: rawData.longitude,
35      },
36    });
37    await image.save();
38    return image;
39  } catch (err) {
40    throw err;
41  }
42}
43
44module.exports = getRaw;
Finally, in the main entry function, slightly modify the previous logic, first search from the database, and then return EXIF or GPS information according to the request:
1const express = require("express");
2const mongoose = require("mongoose");
3const getRaw = require("./lib/getRaw");
4const { Raw } = require("./lib/database");
5
6const app = express();
7const port = 1216;
8
9//连接数据库
10mongoose
11  .connect(process.env.MONGODB_URL || "mongodb://localhost:27017/exif", {
12    useNewUrlParser: true,
13  })
14  .then(() => console.log("Connected to MongoDB"))
15  .catch((err) => console.error("Filed to connect to MongoDB", err));
16
17//以图片url查询数据库，若没有，调用getRaw()
18app.get("/exif", async (req, res) => {
19  const url = req.query.url;
20  const data = await Raw.findOne({ url: url });
21
22  if (!data) {
23    const rawData = await getRaw(url);
24    return res.json(rawData.exif);
25  } else {
26    return res.json(data.exif);
27  }
28});
29
30app.get("/gps", async (req, res) => {
31  const url = req.query.url;
32  const data = await Raw.findOne({ url: url });
33
34  if (!data) {
35    const rawData = await getRaw(url);
36    return res.json(rawData.gps);
37  } else {
38    return res.json(data.gps);
39  }
40});
41
42app.listen(port, () => console.log(`Fetch EXIF is running on port ${port}!`));
At this time, run a MongoDB instance, and then npm start, the interface will run successfully.

Docker Deployment

Because the database needs to be used, it will be troublesome for others to deploy, so I made a Docker Image, and the entire interface can be run through a docker-compose.yml.
1version: '3'
2services:
3  mongo:
4    image: mongo:5.0.17
5    restart: always
6    ports:
7      - 27017:27017
8    networks:
9      - app-network
10    volumes:
11      - ./data:/data/db
12
13  api:
14    image: darmau/fetch-exif:2.0
15    restart: always
16    ports:
17      - 1216:1216
18    networks:
19      - app-network
20    environment:
21      MONGODB_URL: mongodb://mongo:27017/exif
22    depends_on:
23      - mongo
24
25networks:
26  app-network:
27    driver: bridge

评论