Why doesn't the scrollPane become scrollable when I add elements to it?

Question

I am making a GUI in java using swing and I am having some trouble with the scrollPane. As a bit of context, I have a JPanel called leftPanel, on which everything that I am about to show you sits on. This is how it is created:

leftPanel = new JPanel();
leftPanel.setAlignmentX(Component.LEFT_ALIGNMENT);
leftPanel.setBackground(new Color(51, 51, 51));
leftPanel.setPreferredSize(new Dimension(300, 100));

On this panel I then have the following JPanels (they all have something inside, but it's not relevant to the problem):

JPanel topPanel = new JPanel();
topPanel.setPreferredSize(new Dimension(300, 35));
topPanel.setLayout(new BoxLayout(topPanel, BoxLayout.LINE_AXIS));
topPanel.setBackground(new Color(51, 51, 51));

JPanel fenPanel = new JPanel(new FlowLayout());
fenPanel.setPreferredSize(new Dimension(300, 60));
fenPanel.setBackground(new Color(255, 18, 18));

JPanel perftPanel = new JPanel(new FlowLayout(FlowLayout.LEFT));
perftPanel.setPreferredSize(new Dimension(300, 35));
perftPanel.setBackground(new Color(55, 147, 86));

Lastly, I have resultsPanel and scrollPane, which are the reason for this post

resultsPanel = new JPanel();
resultsPanel.setLayout(new BoxLayout(resultsPanel, BoxLayout.PAGE_AXIS));
resultsPanel.setBackground(new Color(51, 51, 51));

JScrollPane scrollPane = new JScrollPane(resultsPanel);
scrollPane.setVerticalScrollBarPolicy(JScrollPane.VERTICAL_SCROLLBAR_ALWAYS);
scrollPane.setHorizontalScrollBarPolicy(JScrollPane.HORIZONTAL_SCROLLBAR_AS_NEEDED);

leftPanel.add(topPanel);
leftPanel.add(fenPanel);
leftPanel.add(perftPanel);
leftPanel.add(scrollPane);

resultsPanel is initially empty, but whenever a button is pressed, some stuff gets printed on there. This is how I do that:

//This thing is in a loop, so more than 1 JLabel is added

JPanel panel = new JPanel(new FlowLayout(FlowLayout.LEFT));
panel.setPreferredSize(new Dimension(300, 20));
panel.setBackground(new Color(51, 51, 51));

JLabel label = null;
if (count == fishCount) {
    label = new JLabel(counterSplit[counterIndex]);
    label.setFont(new Font("Ariel", Font.BOLD, 12));
    label.setBackground(new Color(51, 51, 51));
    label.setForeground(new Color(66, 131, 0));
    label.setOpaque(true);
} else {
    label = new JLabel(counterSplit[counterIndex] + "Stockfish: " + fishCount);
    label.setFont(new Font("Ariel", Font.BOLD, 12));
    label.setBackground(new Color(51, 51, 51));
    label.setForeground(new Color(126, 0, 0));
    label.setOpaque(true);
}

panel.add(label);
resultsPanel.add(panel);

With the code above I didn't specify a preferred size, and for some reason resultsPanel isn't expanding to fit all the content.

If I do add a preferred size, the JLables get printed fine, but the resultsPanel only shows a couple of lines and I can't scroll to see the rest of them, even though they are 100% getting added. I know they are getting printed, because If I make resultsPanel really big I can see the labels there. For some reason though it never becomes scrollable and I don't know what I am doing wrong.

I also wrote this bit of code just to make sure I am not going crazy

JFrame frame = new JFrame("ScrollPane Example");
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

JPanel contentPanel = new JPanel();
contentPanel.setLayout(new BoxLayout(contentPanel, BoxLayout.Y_AXIS));

JScrollPane scrollPane = new JScrollPane(contentPanel);
frame.add(scrollPane);

for (int i = 0; i < 100; i++) {
    contentPanel.add(new JLabel("Label " + i));
}

frame.setSize(300, 200);
frame.setLocationRelativeTo(null);
frame.setVisible(true);

This works exactly as expected and is what I am trying to achieve. If anyone can point out what I am doing wrong, I would greatly appreciate it.