Keep ggplot font size constant in different figure dimensions

I'm in a position that requires me to frequently change the dimensions of my plots via the fig.dim=c(...,...) option in each r chunk.

However, I have a problem that I haven't been able to solve yet. By knitting my document in PDF, if I have two plots with two different specified sizes, their font sizes also differ.

Take this as an example:

'''{r dev="cairo_pdf", fig.dim=c(9,7)}
library(ggplot2)

ggplot(iris, aes(x=Sepal.Length, y=Sepal.Width)) + 
    geom_point() +
    theme_classic() +
    scale_y_continuous(limits = c(2, 5)) +
    scale_x_continuous(limits = c(4, 8))

'''{r dev="cairo_pdf", fig.dim=c(4,3)}
library(ggplot2)

ggplot(iris, aes(x=Sepal.Length, y=Sepal.Width)) + 
  geom_point() +
  theme_classic() +
  scale_y_continuous(limits = c(2, 5)) +
  scale_x_continuous(limits = c(4, 8))

The output will be a document with two plots of different sizes, but the font sizes will also be different:

My question is: is it possible to ensure that, whatever fig.dim I set, the size of all the text in the graphs remains unchanged? Possibly without changing the parameter dev="cairo_pdf", because I use it to use a custom font in my plots.

You could force the output size of the device (cario-pdf), and then manually adjust the font size.

---
output: pdf_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(ggplot2)
```


```{r dev="cairo_pdf", out.height='400pt', out.width='400pt'}
ggplot(iris, aes(x=Sepal.Length, y=Sepal.Width)) + 
    geom_point(size=1.5) +
    theme_classic(base_size = 9) +
    scale_y_continuous(limits = c(2, 5)) +
    scale_x_continuous(limits = c(4, 8))
```


```{r dev="cairo_pdf", out.height='400pt', out.width='200pt'}
ggplot(iris, aes(x=Sepal.Length, y=Sepal.Width)) + 
    geom_point(size=3) +
    theme_classic(base_size = 18) + # double font size to counter half sized out-device
    scale_y_continuous(limits = c(2, 5)) +
    scale_x_continuous(limits = c(4, 8))
```

Note: I use a factor of 2 to make the example clearer (ie 200 * 2 = 400 and 9 * 2 = 18) and also show the same concept with the point size.