textinputformatter not processing clear text flutter

Question

How do I get my Textinputformatter to process _textEditingController.clear() or _textEditingController.text == ''

I have this code:

     TextFormField(
                      enabled: !widget.isLoading,
                      focusNode: _focusNode,
                      keyboardType: TextInputType.phone,
                      controller: _controller,
                      style: const TextStyle(fontSize: Sizes.twenty),
                      decoration: InputDecoration(
                        suffixIcon: IconButton(
                          icon: const Icon(Icons.close_rounded),
                          onPressed: () {
                            _controller.clear();
                          },
                        ),
                        hintText: widget.phonePlaceholder,
                        hintStyle: TextStyle(
                          fontSize: Sizes.twenty,
                          letterSpacing: -0.2,
                          color: Colors.grey[400],
                        ),
                        border: _myOutlineInputBorder(),
                        focusedBorder: _myOutlineInputBorder(),
                      ),
                      inputFormatters: [widget.formatter],
                    )

When I enter some input through the keyboard my inputFormatter react as expected, which means it is called. But the suffix button is called, which is _controller.clear(), nothing happens. I am using LibPhonenumberTextFormatter.

It might that this works as designed. I could read this from the documentation:

Text modification should only be applied when text is being committed by the IME and not on text under composition.

If this is so, then is there a way to clear the textfield such that the input formatter also gets it?