No, the 'stones' won't crumble, or be burned up. Those members of the visible church in this age that are made of 'wood, hay, or straw' are the ones who will get burned up and purged from the people of God, the field and building of God, in the coming judgment of fire.
Compare your words to God's words:
1 Peter 2:1-6 LITV
1) Then laying aside all malice, and all guile, and hypocrisies, and envies, and all evil words,
2) as newborn babes desire the pure soul-nourishing milk, that you may grow by it;
3) if indeed you "tasted" "that the Lord is good;"
4) to whom having drawn near, a living Stone, indeed having been rejected by men, but chosen by God, precious;
5) you also as living stones are being built a spiritual house, a holy priesthood, to offer spiritual sacrifices acceptable to God through Jesus Christ.
6) Because of this, it is also contained in the Scripture: "Behold," I lay in Zion" an elect, "precious Stone," "a Corner-foundation;" "and the one believing in Him shall not be ashamed, never!"
Jesus is building His church, which is His living temple, from living stones.
You would have the temple He builds to be a patchwork of repairs where the flimsy crumbling stones are replaced, the sides shored up.
Are you a living stone being built into a spiritual house, only to be pulled out when you would supposedly disconnect from God?
No, that's now how God speaks of us. You do. But not God.
Much love!