Is your memory reliable?
How do we write about something we don't fully remember? None of us had tape recorders running throughout our childhoods (thank goodness), and our memory of a family event may differ wildly from a sibling's recollection of the same moment. So what do we do? How can we claim something to be 'nonfiction' when we are probably, knowingly or not, making some of it up?
Distinguishing between fact and truth is part of the solution -- you may not remember the exact words your father said, for instance, but you can recall the truth of how he would have spoken -- but another part of the solution is being truthful now about what you remember and what you don't.
If your memory feels hazy, tell the reader. The more transparent you are about the fallibility of your memory, the more honest, and successful, your rendering of the scene can be. "Others may remember it differently, but as I recall..." or "There are as many versions of the story as there were people in the room that day, but what I saw was..."
There's no need to preface every memory this way, but doing it on occasion can give you the freedom you need to assemble the mosaic of memories you have, even if some of those shards are chipped or faint.
Some writers make this acknowledgement in an author's note or preface, such as this one by Elizabeth Rosner in the opening pages of her memoir, Survivor Café:
"Writing this book became a journey navigated by memory as well as its layered and contradictory ellipses. Recounting my own series of trips to Germany with my father, for example, I had to face my own unreliability. Events were frequently refracted through multiple lenses, multiple narrators. And like anyone assembling a puzzle with missing pieces, I wrestled with gaps that could be filled only by way of reconstruction by guessing, by holding on to the filaments of what I mostly knew."
If you are in a similar situation with the crafting of your story, it doesn't mean you cannot write it. It means that you have a creative challenge. And the more honest you are with your reader, and with yourself as the writer, the greater access you will ultimately have to the truth.